To Impute or Not To Impute? Missing Data in Treatment Effect Estimation
Authors: Jeroen Berrevoets
Fergus Imrie
Trent Kyono
James Jordon
Mihaela van der Schaar
Venue: ICLR 2022 Submitted (2022)
Keywords: Causality

Abstract: Missing data is a systemic problem in practical scenarios that causes noise and bias when estimating treatment effects. This makes treatment effect estimation from data with missingness a particularly tricky endeavour. A key reason for this is that standard assumptions on missingness are rendered insufficient due to the presence of an additional variable, treatment, besides the individual and the outcome.  Having a treatment variable introduces additional complexity with respect to why some variables are missing that is overlooked by previous work. In our work we identify a new missingness mechanism, which we term mixed confounded missingness (MCM), where some missingness determines treatment selection and other missingness is determined by treatment selection. Given MCM, we show that naively imputing all data leads to poor performing treatment effects models, as the act of imputation effectively removes information necessary to provide unbiased estimates. However, no imputation at all also leads to biased estimates, as missingness determined by treatment divides the population in distinct subpopulations, where estimates across these populations will be biased. Our solution is selective imputation, where we use insights from MCM to inform precisely which variables should be imputed and which should not. We empirically demonstrate how various learners benefit from selective imputation compared to other solutions for missing data.

Under review as a conference paper at ICLR 2022
TO IMPUTE OR NOT TO IMPUTE? MISSING DATA IN TREATMENT EFFECT ESTIMATION
Anonymous authors Paper under double-blind review
ABSTRACT
Missing data is systemic problem in practical scenarios that causes noise and bias when estimating treatment effects. This makes treatment effect estimation from data with missingness particularly tricky endeavour. A key reason for this is that standard assumptions on missingness are rendered insufficient due to the presence of an additional variable, treatment, besides the individual and the outcome. Hav- ing treatment variable introduces additional complexity with respect to why some variables are missing that is overlooked by previous work. In our work we iden- tify a new missingness mechanism, which we term mixed confounded missingness (MCM), where some missingness determines treatment selection and other miss- ingness is determined by treatment selection. Given MCM, we show that naively imputing all data leads to poor performing treatment effects models, the act of imputation effectively removes information necessary to provide unbiased esti- mates. However, no imputation at all also leads to biased estimates, as missingness determined by treatment divides the population in distinct subpopulations, where estimates across these populations will be biased. Our solution is selective imputa- tion, where we use insights from MCM to inform precisely which variables should be imputed and which should not. We empirically demonstrate how various learn- ers benefit from selective imputation compared to other solutions for missing data.
INTRODUCTION
Treatment effects are arguably the most important estimand in causal inference [Pearl, 2009; Ney- man, 1923; Rubin, 1974]. A reason for this is that a treatment effect lies at the heart of a causal ques- tion. Using causal inference, we try to more explicitly attribute effect to the treatment question, by carefully disentangling the role of the environment. We shall explain with a running example.
Imagine, a job-training program to boost employment. Before sponsoring such a program, legislative body may want to estimate its effect before widespread adoption. Using past data on the program, the body has to rely on causal methods to infer the effect of the training program before formal adoption.
We define effect the difference in outcome when applying the treatment versus not applying the treatment (or any alternative treatment for that matter). Literature on inferring (or predicting) treatment effects is largely concerned with handling selection bias. That is, we identify possible difference between the treated and non-treated subpopulations of the data, seeing that treatment is rarely distributed uniformly across the population. If not accounted for, selection bias will lead to biased estimates. As such, many works focus on novel strategies to handle this bias [Johansson et al., 2016; Alaa van der Schaar, 2017; Shalit et al., 2017; Rubin, 1974; Imbens Rubin, 2015].
In the past, the program was not offered everyone. Those with a job were less likely to be considered. As were those that had been unemployed for a long time. Perhaps age played a role; being less likely to switch careers, older people would benefit less from job-training.
Treatment effect models are adopted in wide range of fields, such as: medicine [Obermeyer & Emanuel, 2016; Alaa et al., 2021; Bica et al., 2021], marketing [Devriendt et al., 2018; Ascarza, 2018: Debaere al., 2019]. or even human resources [Rombaut & Guerry, 2020]. However, these methods have almost exclusively assumed that data is complete. From a practitioner's standpoint, this may not always be the case and, in reality, data is often incomplete [Burton& & Altman, 2004; Lit-

---

Under review as a conference paper at ICLR 2022
tle et al., 2012]. When missingness is not properly accounted for when training these models, their adoption in critical environments, such as medicine, could lead to misguided decisions [Garefa- Laencina et al., 2009; Little & Rubin, 2019]. We define two reasons why missingness may detri- mentally influence treatment effects models. First, when certain covariates are missing when making treatment decision, the fact that these variables
missing may also contribute to selection-bias.
Without knowledge of someone's 's age, it was deemed better to not offer the program. This is understandable, as the high price of the program requires some level of certainty.
A second reason why missingness in the data may have disastrous effects on treatment effects mod- els is that missingness may also be caused by the treatment. Missingness as a result of the treat- ment choice increases the difference between treatment subpopulations, even when there difference to begin with. Effectively, these set of covariates would be very much alike had it not been for their missingness. Adjusting on these missing variables introduces bias to the model.
no
Once an applicant has accepted their offer for job-training, the organisers require some additional information. Imagine, the program's 's registration process asking the applicant's ss current address, the job of their spouse, the amount of children they have, etc. Had they not accepted to participate in the program, they would be less likely to provide this additional information.
The literature proposes two solutions to handle missingness: (i) either we impute the missing vari- ables before subjecting the data to learner [Rubin, 1978; 2004; Little & Rubin, 2019; Kallus al., 2018]; or (ii) we consider a missing variable as another value, and use it directly [Mayer et al., 2020b;a; D' Agostino Jr & Rubin, 2000; Rosenbaum & Rubin, 1984]. We find that imputation effec- tively removes information- i.e. given that missingness may cause treatment-selection,, is much harder to handle selection bias when we are given only part of the total information. However, con- sidering that missingness may be different between treatment subpopulations, it can also introduce bias where there is none to begin with. In this case, not imputing the data will introduce bias.
Contribution. We introduce (and motivate adoption of) a formal description of missingness in data used to estimate treatment effects [Neyman, 1923; Rubin, 1974]. In particular, we find that previous attempts- dating as far back as the 1980s [Rosenbaum & Rubin, 1984, Appendix B] formalising missingness in treatment effects are too general and allow for inaccurate descrip- tions of missingness and its impact. We illustrate why these descriptions are insufficient, and provide an alternative termed mixed confounded missingness (MCM). We argue that MCM a general-purpose missingness mechanism distinct from well-known missingness mechansims such as missing (completely) at random [Little & Rubin, 2019], and refinement of conditional inde- pendence of treatment [Rosenbaum & Rubin, 1984] that should be adopted to describe missing- ness when estimating treatment effects. Furthermore, based on the insights provided by MCM, propose strategy handling missing data in treatment effects, termed selective imputation. Our approach is theoretically motivated and we provide empirical evidence of how methods ben- efit from this approach and demonstrate the harm when missingness is not correctly dealt with.
2 PRELIMINARIES
Estimating causal effects is difficult endeavor, as it requires us to answer a counterfactual question. In particular, when we observe the outcome after applying a treatment on an individual, it impos- sible to also observe that individual's outcome under alternative treatment [Holland, 1986]. As a treatment effect defined as the difference between both outcomes, we are tasked with inferring an estimand which is never observed, which is crucially different from standard supervised learning.
One can estimate causal effects by conducting randomised controlled trials (RCTs) [Fisher, 1925; Neyman, 1923]. However, RCTs are often very expensive, and sometimes considered unethical in a clinical setting [Hellman & Hellman, 1991; Edwards et al., 1999]. However, the alternative we consider in work- estimating effects from observational data comes with its challenges.
Contrasting an RCT study, comparing the subpopulations associated with each treatment in an observational dataset will result in biased estimates. The reason lies in the difference between these subpopulations. If treatment is indeed not assigned randomly, but instead based on an in- dividual's characteristics, then these characteristics more represented in each subpopulation
2

---

Under review as a conference paper at ICLR 2022
a result. If these characteristics also affect the outcome, then the outcomes become biased as the subpopulations are no longer comparable. This phenomenon is often termed selection bias.
Notation. Let X X' S R be the covariates of an individual; let the individual be treated with W E {0, 1]; and let Y E y C R be their observed outcome. Practically, X could be patient with lung-cancer; W = 1 could be chemo-therapy (and W 0 radio- therapy); and Y their tumour size after treatment. We use a subseript, X; to indicate the hth element in X, which means that X; E R.
Assumptions in causal inference. Estimating unbiased treatment effects from observational data has received a lot of attention in recent years. One of the more popular avenues in the literature, is the potential outcomes (POs) framework of causality [Neyman, 1923; Rubin, 1974]. We de- fine the PO of a treatment w E {0, } as (w),, where (()) corresponds to the outcome an individual would have experienced had they been assigned treatment W w. While the stan- dard consistency assumption (see Assum. 1 below) allows us to interpret the observed outcome as the potential outcome of the observed treatment, i.e. Y Y(W); selection bias makes estimat- ing Y(-W) more involved. Countering selection bias is achieved by correctly adjusting for the confounders. In doing so, we make the following assumptions, standard in the PO-framework:
Assumption 1 (Consistency). The observed outcome Y = (W) (w) if W w, for E (0, 1) and = 1, 1.2,...,.N, iie. outcomes in the data correspond to one of the potential outcomes. Assumption 2 (Ignorability). The joint distribution p(X, W,YY satisfies strong ignorability: Y (0), (1) 1 W|X, i.e. the potential outcomes are independent of the treatment, conditioned on X, implying that there are no additional (unobserved) confounders beyond the variables in X.
Assumption 3 (Overlap). The distribution p(X,, W, Y) satisfies overlap: 8 E (0,1) s.t. < PW = z) < 1 8, Vx E X, i.e. each individual has a probability to receive either treatment.
Graphical models and causality. Alternatively to POs, we can express causal relationships as graphical model. In particular, causal relationship is depicted as a directed edge in a directed acyclic graph (DAG) [Pearl, 2009], where a parent node is the cause and the child node is the effect. The ignorability assumption in Assum. 2 is sometimes illustrated in such a graphical model [Richardson & Robins, 2013]. Specifically, the ignorability assumption can be expressed as the DAG shown in Fig. la. Typically the influence of the treatment on the outcome is expressed as a single world intervention graph (SWIG):
[Richardson & Robins, 2013]. We have removed this SWIG- -ath from our figures in order to focus our discussion on the path(s) between X and W.
W
it
Note that the set of DAGs (i.e. the Markov equivalence class) that satisfy Assum. 2 encompasses more than just the DAG in Fig. la. Other DAGs in this equivalence class can just as easily respect Assum. 1 to 3, but they would make no sense. For example, reversing the arrow between X and Y would imply that outcome causes the covariates in the individuals, while still respecting ignorability. Instead, Fig. la is motivated through logical reasoning, where treatment and outcome is caused by the covariates. Throughout the remainder of this paper, we will build (and extend) heavily on Fig. la.
Causal estimands. We now arrive at our two estimands of interest: the average treatment effect (ATE), and the conditional average treatment effect (CATE). Given the notation above, we can define each estimand as follows:
Definition 1 (ATE). The ATE is defined as the population wide difference between a treatment's potential outcomes. Mathematically, we can define the ATE as follows: F(X) == ( Y (O)) Definition 2 (CATE). The CATE is defined as a conditional difference between treatment's 's poten- tial outcomes. Mathematically, we can define the CATE as follows: r(z) = (1) = z).
Missingness. In practice, sample X may be incomplete. For example, a clinician responding to an urgent trauma case may have to select treatment based on incomplete information. In this sce- nario, the incomplete variables are considered missing. To learn from these data we could consider completing this sample through imputation, but missingness in itself may be informative. Perhaps the clinician's 's decision would have been different if they had had complete information. Typically, we detine three mechanisms to describe how a variable ended up to be missing: first is missing completely at random (MCAR), where missingness in one variable is independent of the other vari- ables, second is missing random (MAR), where the missingness in one variable may depend on
'The well-known stable unit treatment value assumption (SUTVA) assumes both no interference and con- sistency [Rubin, 1980]. The equation in our consistency assumption also implics no interference.
3

---

Under review as a conference paper at ICLR 2022
X
X
X
W
Y
X
W
W
Z
W
Z
(a) Ignorability
(b) MCAR
(C) MINAR
(d) MCM
Figure 1: [Fig. Ia] Ignorability as a graphical model. From Richardson Robins [2013], we express ignorability as DAG. For brevity, we dropped the parentheses of Y (w). as well as its accepted "single world intervention path" from w to Y (w); [Figs. 1b to IdJ DGP for missingness mechanisms. Shaded nodes indicate nodes that relate to missing variables, white nodes relate to treatment effects. In Fig. lb and Fig. Ic we illustrate MCAR and M(N)AR, respectively. In Fig. Id we illustrate MCM. Unique to MCM is to allow for treatment to cause missingness (through XX and "Z), while also allowing for treatment to be caused by missingness (corresponding to X and 2*).
the other (observed) variables, and the third is missing not at random (MNAR), where missingness is typically assumed to be caused by variables outside the observed covariates [van Buuren, 2018; Rubin, 1976; Little & Rubin, 2019]. Generally, MCAR attributed to noise in data-collection.
We indicate the missing data in X with variable Z E {1,*]á where Z: = * if X; is missing, and Z, = 1 if X, is observed. Having I X, (w), corresponds with MCAR; and Z Y X, but Z L Y (w) X, corresponds with M(N)AR. In our paper, denote the complete (but unobserved) sample as X, and the incomplete (but observed) sample as X Z @ X E {+, where we define C as the element-wise product and we take X, * = *, i.. if X, is unobserved it equals in X.
Note that none of the existing missingness mechanisms take the treatment W into account as stan- dard, since they span a broader literature beyond treatment effects. We have illustrated MCAR and M(N)AR as DAGs in Fig. lb and Fig. 1c. respectively, where we have included arrows from X and Z to W. This corresponds observed; the alternative where X is causing W would lead to a confounded setting, which is com- monly assumed not be the case in the literature. Note that these DAGs respect the conditional in- dependent statements assumed in their respective missingness mechanisms. As we have noted in our introduction, we could either impute a missing value X,j with an estimate thereof, denoted Xij. such that we can predict treatment effect from data with imputed samples (i.e. learn F(X) 7 (X)); we could predict treatment effects from data with missingness directly (i.e. learn F(X) or T(X)).
the situation where treatment decided based on what is actually
)); or
MIXED CONFOUNDED MISSINGNESS (MCM)
Although MCAR and M(N)AR may capture some settings that include a treatment, we argue that they do not capture all of them. As we will explain below, some missingness may be caused by treatment, while other missingness may cause treatment. This has important consequences as miss- ingness is now a mixture of confounding and non-confounding elements, leading us to term our proposal mixed confounded missingness (MCM). We have illustrated MCM as DAG in Fig. 1d. In this section, we will explain why there are no other arrows included in MCM, i.e. motivate that it is complete and general. Then, we compare MCM to previous proposals for missingness in CATE.
Like MCAR and M(N)AR, MCM describes the interactions between Z and the remaining variables, X, W, Y Y(w), (and X). While MCM is fully defined by Fig. 1d, the definition rests the assump- tion that 2 can be split into two distinct factors, 2 and *7, where the former captures missingness causing treatment, and the latter missingness caused by treatment. Formally, with MCM assume:
Definition 3 (Missingness factors in MCM). We assume there exists a partition of Z = {2t, ?2) s.t. Z² 1 W,W,X, further implying that X 1 'X|W, X Z" 1 X W and X* 1 Z W.
MCM is exhaustive. When each possible causal path is taken into account with respect to the missingness variables, we say that MCM is exhaustive. Being exhaustive is important, as through exhaustiveness can we argue that MCM covers all possible scenarios that can lead to missing data with treatments. Having five variables (X, *X, X", W, and Y) be fully connected to Z and Z should result in 20 distinct paths (four for each of the five variables). However, we only count 6
4

---

Under review as a conference paper at ICLR 2022
X
Z
W
Z
(a)
(b)
(c)
(d)
Figure 2: Arrows that are not included in MCM. Considering each possible direct arrow from, and to, the missingness variables (Z and Z') should result in 20 arrows, as there are five remaining variables. Excluding the paths that are included, and the paths across missingness indicators, we are left with 10 paths that are seemingly missing from our definition. From Figs. 2a to 2d, we depict each missing arrow in function of their related variable: X, then Y (w)), W, and then X" and *X.
direct paths in Fig, 1d that are connected to either Z or Z' leaving 14 paths unaccounted for. With def. 3, we can exclude paths across missingness factors (i.e. the paths that directly connect Z and "X, and those that directly connect "Z to X*), reducing the number of unaccounted paths to 10.
We will now argue why these 10 paths (in Fig. 2) are not included in MCM. First, we discuss Fig. 2a. Having Z missing. implying that is dependent on whatever dataset an individual included in. Having this dependence would imply that X. representing the fully observed- true -set of covariates, would be different across datasets, despite representing the same individual. Furthermore, including the paths (and similarly for 2²), which violates the DAG structure.
Z allows covariates change depending on what other variables are
in Fig. 2a introduces a cycle: X
In Fig. 2b we immediately observe cycles, meaning that only two of the four presented arrows exist simultaneously. Let us first consider the paths where Y(w) is causing missingness, i.e. In the potential outcomes setting, Y (w) is topologically last in the DAG; take
Y- Fig. la, where inclusion upon of the single world intervention path, that no variable is caused by Y (w). Having (w)) as the final observation makes sense; once we observe the outcome from a treatment. the covariate observations are left untouched. The absence of the remaining two arrows,
W Y we clearly see
--- Z is similarly argued as the absence of the arrows in Fig. 2a. Namely, if Z were directly influence outcome, then an individual represented in two different datasets- with different missing variables -would have conficcing outcomes. Clearly, one person can only have one outcome [Holland, 1986), i.e. the arrows in Fig. 2b cannot exist.
Next, Fig. 2c, illustrating the existing arrows, reversed. Besides these arrows resulting in cycles- specifically, ( -they also violate our definition for 7 and Z In particular, the raison d' d'être for these distinct factors is precisely their respective directed paths from and to W.
Lastly, we consider Fig. 2d depicting the reverse arrows from * to 'Z, and similarly from X to Z'. Here too, besides the obvious cycles,
and similarly for X and Z', X is determin- istically defined an element-wise product between Z and X. This relationship is unambiguous. If indeed ZZ and Z are caused by the fully observed covariates X, it seems almost silly to consider them to be also caused by the partially observed covariates X, which represents almost exactly the same entity. In fact, their only difference is completely captured in Z. With this, we safely remove the arrows in Fig. 2, resulting in MCM (with a total of 6 arrows in Z) to be exhaustive.
Missing data and existing assumptions. Having argued each arrow in Fig. 1d, will now relate MCM to previous descriptions of missingness in the treatment effects setting. Introduced in Rosen- baum & Rubin [1984] and further investigated in Matte [2009]: Mayer al. [2020b] and Blake et al. [2020], we find that typically the ignorability assumption in the treatment effects literature (that is, assum. 2) is extended the setting with missingness by simply replacing the condition,
Y(O), TL WIX,2, Z,
which translates to "unconfoundedness despite missingness" #2, To verify Eq. (1), typically the above unconfoundedness assumption is combined with one of the two assumptions below [Mattei, 2009],
2 NNte that, we do not have to explicitly include Z in the condition (see for example Mayer et al. [2020b])
X is related in deterministic way Z. However, we chose include Z in these conditions, as they are
5

---

Under review as a conference paper at ICLR 2022
CIT: W 1 XXXX Z
CIO: Y(0)Y(() 1 XXXX Z.
(2, 3)
where CIT in Eq. (2) stands for conditional independence of treatment, and CIO in Eg. (3) stands for conditional independence of outcome [Mayer et al., 2020b]. Essentially, CIT and CIO assume there no additional information in the fully observed X compared to the observed X and Z to predict treatment and outcome, respectively- i.e. adjustment for X (and Z) still warrants ignorability.
The assumptions depicted in Eqs. (2) and (3) can be considered logical consequence of the assump- tion in Eq. (1). In particular, for the potential outcomes to be ignorable from treatment, as is assumed through Eq. (1), there cannot exist direct arrows beyond those in X' from X to either W (which is implied by Eq. (2)), or Y (w) (which is implied by Eq. (3)). Violating CIT and CIO would mean that there has to be such a variable (i.e. dimension) in X, that is not represented in the covariate space of X, that has direct link to either W, or Y (2)). both. Such link is in direct conflict of Eq. (I).
At first glance, CIT and CIO may seem acceptable assumptions. However, we identify two ma- jor issues concerning these assumptions: (i) despite them further specifying the consequences of ignorability despite missingness (cfr. Eq. (I)) CIT and CIO are too general, allowing too many different (mostly completely unrealistic) missingness mechanisms without violating Eq. (1) nor CIT or CIO; and (ii) they do not allow for treatments to cause missingness, which we have argued above an important consideration in treatment effects. Furthermore, problem (ii) hints at larger un- derlying problem regarding missingness in treatment effects, which we shall discuss in Section 4.
Given the variables in Eq. (1) and CIT/CIO (X, X, Z, (w)), while keeping
fixed, we can generate 42 DAGs which all respect these assumptions. We have included these DAGs in Appendix D. Note that each of these 42 DAGs respects the independence statements in Eq. (1) and those in either Eq. (2) or Eq. (3). Evaluating these DAGs using the same criteria as we have for MCM leads to only realistic DAG, which turns out to be special case of MCM. Specifically, when removing the consideration of Z- treatment causing missingness -we arrive at the one DAG that realistic. Without explicit consideration of treatment choices that influence missingness (i.e. splitting Z in Z" and "Z), we learn that MCM fits nicely within CIT (and consequentially Eq. (1)). Furthermore, this one DAG corresponds exactly to how we introduced M(N)AR in Fig. 1c.
4 SELECTIVE IMPUTATION
Having introduced MCM, we now explain why missing data in treatment effects should not be brushed over lightly. We discuss why the two common approaches to handle missing data (recall that typically missing data is either imputed, or kept as is) are not equipped to deal appropriately with missing data that interacts with treatment. Here we offer an alternative to these two approaches.
Why naive approaches don't work. First, we have to discuss what can go wrong when naively dealing with missing data. To aid our discussion, we simpli- fied MCM in Fig. 3, which merges Z² into X and * into *X. Note that this is equivalent to Fig. ld as the link between the missingness indicator and the observed covariates is complctely deterministic. One could, with complete certainty, derive the missing- ness indicator from the observed covariates. In fact, the only way of obtaining Z is do exactly that.
X
W
Figure 3: Ignorability of the covariate- factors in MCM. Above DAG depicts MCM, where have included Z and *7 in the factors X and "X, respectively. Be- sides MCM, we have also included "X' as an additional node, meaning the imputed co- variate set stemming directly from X
Imputing all data, i.e. the first naive approach, has the objective of recovering X from X, as accurately as possible. Given that (w), is a direct function of the fully observed set of covariates, X, and not the partially observed set, X, regressing Y (w) on
related to the DAG presented in Fig. 1d, where we have also included 2. Later in our paper, we will remove from DAGs and subsequently our conditions, in order to focus the relationships with X directly.

---

Under review as a conference paper at ICLR 2022
X should result in better estimates than X. Contrasting supervised learning, however, the treat- ment effects literature is not concerned just with better estimates. Indeed, in treatment effects, the target is different from the observed outcomes in the dataset. In particular, to estimate treat- ment effects, one has to account for selection bias [Shalit et al., 2017], which is expressed in the path from X to W (see for example, Fig. 1a). In our setting, this is more complicated as there exists no direct path. The path from tion 3, treatment is decided on an individual's covariates, but also the missingness in them. As such, imputing away these missing data will result in information loss with respect to selection bias, making it all the more harder to counteract the bias when predicting treatment effect.
W is mediated by X As explained in Sec-
Should we then impute no data, i.e. the second naive approach? Not imputing data certainly solves the problem of information loss. However, it introduces another problem. We learn from Sec- tion 3, that in order to estimate treatment effects from data, one has to assume ignorability despite missingness (see Eq. (I)). However, from Fig. 3 we learn that (w)) 1 W|X*, XX i.e. the po- tential outcomes are not conditionally independent from treatment at all. We find the reason in 'X, which- due to being determined by treatment composse a collider structure. A property of a collider structure is that is does not bias treatment selection, but when conditioned upon, it does. If imputing data causes problems, and not imputing data causes problems, what then should we do?
Selective imputation. While solutions could vary across different datasets, we provide general strategy to handle missing data in a treatment effects setting, termed selective imputation. Given the aforementioned problems, it is clear that both imputing and not imputing has flaws and mer- its. In fact, the problem the one, is solved by the other. Consider again Fig. 3, where we have included *X" in the MCM-DAG. From evaluating Fig. 3 with this inclusion we learn that,
Y(0), WIX".X.
arriving once more at a conditional independence between W and (w). Notice that, the indepen- dence statement in Eq. (1) conditions on the same covariate information (i.e. all is included from X), the only thing changed, that we have to "forget" that some data in 'X may be missing as the missingness associated with "X is determined by the treatment- property we need to account for.
The role of imputation. From above discussion, we learn that missingness is what differentiates treatment- sutiinenttuupopplaatiionsss in "X. The following theorem shows what exactly imputation needs to accomplish, in order to consider Eq. (4) to be true (proof of theorem can be found in Appendix A).
Theorem 1. Suppose the graph structure in Fig. 3. and W and "X are independent, then ignorability defined as Y (0), (1) 1 W|X', X holds.
From theorem we learn that, in this context, a good imputation strategy should aim to make "X independent of the treatment, such that ignorability may hold. The intuition behind this is that through the act of (proper) imputation we effectively balance 'X, making the treatment populations indistinguishable from each other [Shalit et al., 2017]. The better the imputation is, the closer to X the covariates in "X" will become, reducing influence from W. If our imputation is of poor quality, a model may recover information about the originally missing variables, i.c. information on treatment is retained X still correlates with W, which allows bias to creep into our models.
5 EXPERIMENTS
We turn now to empirically validating that selective imputation, i.e. imputing only some parts of the covariate space, results in better treatment effect predictions. Note that good prediction performance on the outcomes present in the dataset is not considered better in the treatment effects literature [Alaa & van der Schaar, 2017; Shalit et al., 2017; 77 Johansson et al., 2016], which we explain in detail below.
Data. Common in the treatment effects literature is the use of synthetic datasets. The reason why we have to rely on synthetic data lies at the core of the problem: in any real-world setting, the counterfactual is unobserved. If the counterfactual is indeed unobserved, we cannot evaluate treat- ment effects models on how well they predict the counterfactual. As such, with synthetic data simulate both potential outcomes, effectively observing the counterfactual for evaluation.
7

---

Under review as a conference paper at ICLR 2022
Table 1: ATE results. For each setting we report an MSE between the predicted ATE and the simulated (ground truth) ATE on 10 different train and test sets using 10 differently sampled simu- lations and averaged the results, standard deviation is in brackets. Imputing only "X ("Selective"), consistently performs best across different ATE prediction methodologies and treatments. Lower is better, our proposal is shaded
Impute
T-Learner
Doubly Rob.
X-Learner
description
covariates
All
{X*, X
0.0951 (.010)
0.0651 (.008)
0.0472 (.009)
RSW
Nothing
[0]
0.0642 (.027)
0.0902 (.018)
0.0726 (.024)
MS
Selective
[X]
0.0403 (.014)
0.0381 (.009)
0.0309 (.014)
Sel. Complement
{X*}
0.0931 (.026)
0.0902 (.019)
0.0984 (.040)
Next to counterfactual evaluation, there is an additional reason why we have to rely on simulation: we have to be able to control the missingness mechanism. A simulation is the only way to test whether treatment effects models are affected by MCM. Our simulation is described in Appendix B.
For each experiment, we sample 10 different simulated datasets, from which we sample 10 random train and test sets, for each treatment effects method. As our finding holds both for ATE as well as CATE (in finite settings [Alaa & van der Schaar, 2018J), we test on both scenarios. Each simulated dataset contains 10k samples. The datasets span 20 dimensions (with factors of equal size), and missingness rate of 0.3. We have performed multiple ablations these values in Appendix C.
Imputation. We define four different scenarios: either we impute missing variables across () all the variables (indicated "All"); (ii) none of the variables ("Nothing"); (iii) only (iv) only the variables
the variables in "X ("Selective"); ment"). Following Section 4, scenario (iii) (imputing only "X) should yield the best re- sults given that bias is removed, while information with respect to treatment selection is tained. Imputation is performed using MICE [van Buuren & OuddhhooOO 2011].
X ("Sel. Comple-
Models. In our experiments, we evaluate the performance of three classes of learners in the imputa- tion scenarios described above: T-learner, Doubly robust (DR) learner, and X-learner. We pair each treatment effects method with XGBooSt due to its ability naturally handle missing values. We refer to Künzel et al. [2019] or Curth & van der Schaar [2021] for an overview of various learners.
5.1 AVERAGE TREATMENT EFFECTS (ATE)
Objective. A first estimand to consider is the average treatment effect (defined in def. 1). While the ATE has been considered since the 1970's [Rubin, 1974), it is still important causal estimand today. It can be argued that ATES suffer more from selection bias than the conditional ATE, as the average is computed over the entire dataset, unlike the CATE [Alaa & van der Schaar, 2018]. In ATE, adjustment plays an important role and is usually achieved by placing non-uniform weights on each element in the dataset when computing an average. These weights often take the form of inverse propensity weights (IPW), (p(W 1 used by, for example, DR-learner and X-learerr.
Results. Consider Table where we reported the mean squared error (MSE) between an estimated ATE and the ground-truie truth ATE across ten folds, for ten differently sampled simulations (thus span- ning 100 trained learners of each type). Given three popular ATE estimation strategies, we find that imputing "X" significantly performs better across all methods, confirming the insights in Section 4.
5.2 CONDITIONAL AVERAGE TREATMENT EFFECTS (CATE)
Objective. Similar to Section 5.1, we will now evaluate how CATE-leamers react to MCM. Specifi- cally, we subject the same treatment effects learners to the same scenarios have in Section 5.1, and evaluate their CATT-predictions using the PEHE metric described above. As was the case for ATE experiments, the chosen learners represent wide range of different methodologies.
8

---

Under review as a conference paper at ICLR 2022
Table 2: CATE results. We report PEHE on 10 different random train and test sets using 10 differently sampled simulations and averaged the results, standard deviation is in brackets. Imputing only *X while keeping X is (marked as "Selective" below), consistently performs best across learners and treatments. Lower is better, our proposal is shaded
Impute
T-learner
Doubly rob.
X-learner
description
covariates
All
[X', X
0.7603 (0.051)
1.3674 (1.731)
0.6149 (0.063)
PEHE
Nothing
0
0.6906 (0.072)
0,9409 (1.94)
0.3027 (0.085)
Selective
("X]
0.4605 (0.045)
0.2042 (0.224)
0.2116 (0.032)
Sel. Complement
[X*]
0.9158 (0.064)
4.3657 (8.823)
0.4912 (0.109)
PEHEw
All
[X', }
0.7371 I (0.081)
1.2083 (1.610)
0.6272 (0.070)
Nothing
[
0.7015 (0.100)
0.8130 (1.287)
0.2907 (0.107)
Selective
{ ['X]
0.5720 (0.079)
0.1787 (0.202)
0.2556 (0.062)
Sel. Complement
[X*}
0.9351 (0.120)
4.2306 (9.056)
0.5198 (0.156)
PEHEW-
All
[X*, "
0.7726 (0.055)
1,4419 (1.802)
0.6097 (0.068)
Nothing
[0]
0.6881 (0.090)
0.9973 (2.317)
0.3091 (0.098)
Selective
{ [X}
0.4097 (0.045)
0.2169 (0.236)
0.1915 (0.036)
Sel. Complement
[X*}
0.9183 (0.075)
4.4322 (8.749)
0.4803 (0.130)
Different from evaluation in ATE, the typical evaluation metric CATE the precision in estimat- ing heterogeneous treafment effects (PEHE), defined as, Ex (x((((()) - f
f(X))² in Hill [2011], where f is a model's prediction. Naturally, there is a parallel with the MSE we used in Section 5,1, where PEHE essentially corresponds to the MSE over the predicted vector of CATEs in holdout test-set.
Results. Reported Table we find results for various CATE-learness across ten differently sam- pled simulations, each evaluated with ten different train and test sets we have for our ATE in Section 5.1. Given these results, we empirically confirm that one should impute cautiously as imput- ing all data, no data, wrong data, consistently performs worse than what suggest: impute only "X More settings and configurations are reported in Appendix C, all of which confirm our findings.
DISCUSSION
Estimating treatment effects is becoming more important in many practical settings. The adoption of these methods is largely the result of great academic effort to further push the boundaries of these methods' abilities. While practical adoption indeed point in favour of causal methods, comes with a significant downside; unlike in academia, datasets used in practice are often victim to many imperfections. In our paper, we investigated one imperfection in particular: missing variables.
Missing variables in treatment effect settings behave differently from other settings; they should be treated differently as a result. our paper, we argue that some missingness can informative of the treatment, i.e. just like certain characteristics of an individual may determine their treatment, so too can the absence of certain measurments determine treatment. If one should impute these variables, that information is lost and can no longer be used to counter any resulting selection-bias.
We argue further that we should not leave all variables unimputed. When there are variables that only missing because individual was given particular treatment, then these missing vari- ables are informative of the treatment and result in a covariate shift between treatment groups.
We believe the key finding of paper is summarised follows: more care and thought should be put into imputing missing data when estimating treatment effects. We confirm intuitively, the- oretically, and empirically that selectively imputing missing variables can improve our ability to estimate treatment effect, while wrong imputation can lead to poorly modelled treatment effects.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
All our code is provide in the supplementary materials; most of it is based on open- source code such causalml, econml, and sklearn. GAIN is implemented using from Yoon
the authors' online code (efr. https://gi al. [2018]). Code for our data-generating process shown in Appendix B, and is provided as separate files in the supplementary materials. We provide notebooks produce the results reported in the main text as well as those reported Appendix C.
REFERENCES
Ahmed Alaa and Mihaela van der Schaar. Limits of estimating heterogeneous treatment effects: Guidelines for practical algorithm design. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 129 129-138. PMLR, 10-15 Jul 2018. URL https: pttpp/ercceee.........
mlr. www.ppeessvv00 v80 awwalllaaea Ba html.
Ahmed M. Alaa and Mihaela van der Schaar. Bayesian inference of individualized treatment effects using multi-task gaussian processes. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fer- gus, S. Vishwanathan, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: pttppppocccenn..eeeeeppp neurips.er..mm cc/ paper/2017/E5 1tt.0////ll/////////////// 6a509a f951 Oea 6acb021 wtwp://99bb---aer.ppepom
Ahmed M Alaa, Deepti Gurdasani, Adrian L Harris, Jem Rashbass, and Mihaela van der Schaar. Machine learning guide the use of adjuvant therapies for breast cancer. Nature Machine Intel- ligence, 3(8):716-726, 2021.
Eva Ascarza. Retention futility: Targeting high-risk customers might be ineffective. Journal of Marketing Research, 55(1):80-98, 2018.
Ioana Bica, Ahmed M Alaa, Craig Lambert, and Mihaela Van Der Schaar. From real-world patient data to individualized treatment effects using machine learning: Current and future methods to address underlying challenges. Clinical Pharmacology & Therapeutics, 109(1):87-100, 2021.
Helen A Blake, Clémence Leyrat, Kathryn E Mansfield, Laurie A Tomlinson, James Carpenter, and Elizabeth J Williamson. Estimating treatment effects with partially observed covariates using outcome regression with missing indicators. Biometrical Journal, 62(2):428 443, 2020.
A Burton and DG Altman. Missing covariate data within cancer prognostic studies: review of current reporting and proposed guidelines. British Jritishjournal of cancer, 91(1):4-8, 2004.
Alicia Curth and Mihaela van der Schaar. Nonparametric estimation of heterogeneous treatment effects: From theory to learning algorithms. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 1810-1818. PMLR, 13-15 Apr 2021. URL wwt.tttpp/.cc...... /tttps://proceee proceed ngs ml w...pp.es//....................e v1 30/ cu rth21a html.
Ralph B Agoostno Jr and Donald B Rubin. Estimating and using propensity scores with partially missing data. Journal of the American Statistical Association, 95(451):749-759 2000.
Steven Debaere, Floris Devriendt, Johanna Brunneder, Wouter Verbeke, Tom De Ruyck, and Kristof Coussement. Reducing inferior member community participation using uplift modeling: Evi- dence from field experiment. Decision Support Systems, 123:113077, 2019.
Floris Devriendt, Darie Moldovan, and Wouter Verbeke. A literature survey and experimental eval- uation of the state-of-the-art in uplift modeling: A stepping stone toward the development of prescriptive analytics. Big data, 6(1):13-41, 2018.
Sarah JL Edwards, David A Braunholtz, Richard J Lilford, and Andrew J Stevens. Ethical issues in the design and conduct of cluster randomised controlled trials. BMJ. 318(7195):1407-1449, 1999.
10

---

Under review as a conference paper at ICLR 2022
Ronald A Fisher. Statistical methods for research workers. Oliver and Boyd, 1 edition, 1925.
Pedro Gareia-Laencina, José-Luis Sancho-Gómez, and Aníbal R. Figueiras- Vidal.. Pattern classi- fication with missing data: review. Neural Computing and Applications, 19:263-282, 2009.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sher- jil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL 5ca3e 22661
newriiss.. ctlas.cc/ppee wttppppppee//1//111ee. pdf.
Samuel Hellman and Deborah S. Hellman. Of mice but not men. New England Journal of Medicine, 324(22)1585-1588 1991. doi: 10.1066/EEMM11110555022444
Jennifer Hill. Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20(1):217-26) 2011.
Paul Holland. Statisties and causal inference. Journal of the American statistical Association, (396):945-90) 1986.
Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sci- ences. Cambridge University Press, 2015.
Fredrik Johansson, Uri Shalit, and David Sontag. Learning representations for counterfactual ference. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning. volume 48 of Proceedings of Machine Learn- ing Research, pp. 3020-3029, New York. New York, USA, 20-22 Jun 2016. PMLR. URL https: http::s/pppoceediing
mlr press /v48 / www.soaaassnnoo.o 6 htm..
Nathan Kallus, Xiaojic Mao, and Madeleine Udell. Causal inference with noisy and missing covariates via matrix factorization.
Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https: ptttps::/proceeeeeee http://prceeedngceeeeeep neurips...... cwt.rrccccc///ccc.eeecooo paper/2018. /ile//el/
pdf.
Sören R Künzel, Jasjeet Sekhon, Peter J Bickel, and Bin Yu. Metaleamers for estimating heteroge- neous treatment effects using machine learning. Proceedings of the national academy of sciences, 116(10)4416-44655 2019.
Roderick J Little, Ralph D' Agostino, Michael L Cohen, Kay Dickersin, Scott S Emerson, John T Farrar, Constantine Frangakis, Joseph W Hogan, Geert Molenberghs, Susan Murphy, et al. The prevention and treatment of missing data in clinical trials. New England Journal of Medicine, 367 314):1335-1360, 2012.
Roderick JA Little and Donald B Rubin. Statistical analysis with missing data, volume 793. John Wiley & Sons, 2019.
Alessandra Mattei. Estimating and using propensity score in presence of missing background data: an application to assess the impact of childbearing on wellbeing. Statistical Methods and Appli- cations, ((2))2577273, 2009.
Inke Mayer, Julie Josse, Félix Raimundo, and Cean-Philippp Vert. MissDeepCauusl: Causal infer- from incomplete data using deep latent variable models. arXiv preprint arXiv:2002.10837, 2020a.
Inke Mayer, Erik Sverdrup, Tobias Gauss, Jean- Denis Moyer, Stefan Wager, and Julie Josse. Doubly robust treatment effect estimation with missing attributes. The Annals of Applied Statistics, 14(3): 1409-1431, 2020b.
Jersey Neyman. Sur les applications de théorie des probabilités aux experiences agricoles: Essai des principes. Roczniki Nauk Roiniczych, 10:1-51, 1923.

---

Under review as a conference paper at ICLR 2022
Ziad Obermeyer and Ezekiel J Emanuel. Predicting the future big data, machine learning, and clinical medicine. The New England journal of medicine, 375(13): 2016.
Judea Pearl. Causality. Cambridge university press, 2009.
Thomas S Richardson and James M Robins. Single world intervention graphs (swigs): A unification of the counterfactual and graphical approaches to causality. Center the Statistics and the Social Sciences, University of Washington Series. Working Paper, 128(30):2013, 2013.
Evy Rombaut and Marie-Anne Guerry. The effectiveness of employee retention through an uplift modeling approach. International Journal of Manpower, 2020.
Paul R Rosenbaum and Donald B Rubin. Reducing bias in observational studies using subclassifi- cation the propensity score. Journal of the American statistical Association, 79(387):516-522) 1984.
Donald B Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies.
Journal of educational Psychology, 66(5):688, 1974.
Donald B Rubin. Inference and missing data. Biometrika, 63(3):581-592, 1976.
Donald B Rubin. Multiple imputations in sample surveys-a phenomenologica bayesian approach to nonresponse. In Proceedings of the survey research methods section of the American Statistical Association, volume 1, pp. 20-34. American Statistical Association, 1978.
Donald B. Rubin. Comment on "randomization analysis of experimental data: The fisher ran- domization test". Journal of the American Statistical Association, 75(371):591, 1980. doi: 10.2307/2287653. URL UtLhttss:/ /da/ .ttee/orrg11...0
Donald B Rubin. Multiple imputation for nonresponse in surveys, volume 81. John Wiley & Sons, 2004.
Uri Shalit, Fredrik D. Johansson, and David Sontag. Estimating individual treatment effect: general- ization bounds and algorithms. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research. pp. 3076-3085. PMLR, 06-11 Aug 2017. URL https: http///prceeee...eeeeee//oo.. mmrr.
press/v70/ 20t0/t/haltt7aaammmm html..
Stef van Buuren. Flexible Imputation of Missing Data. Chapman & Hall/CRC Interdisciplinary Statistics. CRC Press LLC., 2018. ISBN 9781 138588388.
Stef van Buuren and Catharina equations in Journal of Statistical Software, 45, 12 2011. doi: 10. 118637/jss.0045.000
MICE: Multivariate imputation by chained
Jinsung Yoon, James Jordon, and Mihaela van der Schaar. GAIN: Missing data imputation using generative adversarial nets. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pp. press/v80/yoon html.
PMLR, 10-15 Jul 2018. URL https: https::/proceee: http:///epopog/s/seeeeee/e// .mlr..... mmlr.
12

---

Under review as a conference paper at ICLR 2022
A PROOF OF THEOREM
Theorem 1. Suppose the graph structure in Fig. 3, and W and "X are independent, then ignorability defined as Y (0), (()) 1 WIX*, "X holds.
Proof. Consider p(Y = W = introducing and using the definition the conditional density, we have
4), which we write as p(y. w/2, i). Then by
f plu, w, 2)
ply 1 2)
(5)
[ plu. w. 2,
We can rewrite ply, w, x, ū, using the graph structure
ply w,, ã, (fls::))p((/z))pp()l)))
w)p(x)
(6)
Noting that X not a collider of W and "X, and by independence of W and "X we have that p(ala, w) p()(2))
By Lemma 1, X and "X conditionally independent given X. Therefore p(z(x)p(((z) p(ž, flx). Similarly, X d-separates Y and the pair (X+, "X) and thus,
p(yla)p((la)p(tlee) ply ā, 2|z)).
Substituting (7) and (8) into (6), gives
ply, H. + (w))))((()))))))) ply, w, x. R, it)
We then note that X is not collider of W and 'X so that,
p(w|2) (
(10)
Substituting this all into (5) gives us
/ ply. x. a k (elp x)p())
ply, w|ē,
((ū,))
plu. 3, ())(w, p x)
(11)
p(ylū,
which proves the result.
Lemma 1. Suppose the setting Theorem 1. Then X and X are conditionally independent given
X.
Proof. By appealing to the DAG to write the full joint distribution of p(x, .7, w. T. .) we have
f(x))
p( (
F.w
( (p()))))
(12)
=p(5" (()p)))(())))))))))))))))))))))) [ p(wlā²)
-(("" (x)p(((>)
13

---

Under review as a conference paper at ICLR 2022
B DATA
Standard in treatment effects literature, proper empirical validation requires synthetie data. Here we describe our synthetic setup, and how we sample from it precisely (clarifying with Python code).
IStep 1] Sample X. In order for imputation to make sense, there has to exist some correlation between variables in X, such sampling from a standard normal with the identity matrix covariance matrix, will not suffice. As such, we sample from a normal distribution with a random (positive semidefinite covariance matrix, spanning 20 dimensions:
import numpy as np
def seneeratecovaaiaates (d, n)
assert assert < n
A = random. rand (d, d)
cov np. dot (A, A.transpose )))
10
np.random.c wwwttvarriate_nnraaaa (np. zeros (d), cov, size-n)
X /= (x) ())
return
above code we have d=20 the dimension count, and n=10000 the sample size. our experi- ments we sample 10 different train and test sets from the data to calculate per simulation descriptives.
[Step 2] Sample Z². From we generate 7 as follows:
14
h gheet_ goodd_bbrddrr X :?:,:7_di argsort (exis-1) [:,
15
-nnt (ap. max (int (np. round (amount of missingness z_dim) ) 1)1)
o_o list
[tiigeestbborerrll) for i, in enumerate [X]:,
z (dimm))) )
17
2_out mp.array (z_out) asstye (int)
18
out
abs (2_out-1)
In above code have two main parameters: dim=10 indicating the amount of variables in 7*, and amount_off mm ss maiingness=--.. indicating the fraction of the data that is missing. in In 14 above, calculate threshold, when the value of X; above this threshold, that variable is missing. This threshold corresponds to the amount of dimensions that need to be missing in order respect amount_of.mm ss wwwissinnneenn miingness. Notice that Z now corresponds X z_dim).
[Step 3) Sample W. We make very explicit that treatment choices depend on Z
[] for =_ in 2_out:
21
if d
22
w 0
23
elif 0 in a_dl:int (np. fleer (z_dim/2))]
W =1
25
clse:
26
w np. random. binomial (1, .5)
27
W. apapped (w)
W ap. (W)
[Step 4) Sample Z. Sampling Z. which depends W in such a way that the arrow is identifiable, requires interaction between X and W For this we sample two random vectors (theta.2 a_a-in_n.. and theta.z.in.l) and let those interact with decide Z.. As was the case with Z also calculate the amount of variables that should be affected (calculated in 1n 29-32). Note that the dim count this setting only corresponds approximately to the amount. of m ss ingnes..
29
import scipy
30
Note that in the non-binary setting, this interaction may necessary.
14

---

Under review as a conference paper at ICLR 2022
dim count np. round (amount of missingness
dii)
32
dim_count np.mas max (din_count, 1))
33
dim_count np.min (dim_count, int ((d d_mm) / 2
34
dim_count - int (dim_count)
theta_z_ing np. fulf (din_ccunt, scipy. stats stapy..sttt..nom.pp...
amount_of _isss__ss_issssneessss
37
theta_z_in_ = np. full (dim_count, scipy. morats..orm.pp... norm.ppf (1
amount_of moonn__es_dssingness))
2_in np.zeros( (n, d ?_dim))
40
for i, in enumerate (V_in):
41
x - x[i, __di::2_dim+din_com it Wiil:
42
_iini, -dim count:]
X[;, z dim:z dii:z_dimtdimmmmmmmddiim www.ddiiimoouuiioo mean
axis=0)) (theta_z_in__ x.std(exis==))
44
clsc:
45
2 iin :din_count] = (x XI:, dddimma__dimddi__ommm mean (
axis=0) acheta_e_in_ x.xtd(exis=())
2nn np. (2 in-1)
[Step 5] Sample (ww). Generating outcomes done simply by sampling two linear functions.
47
def
generate out comes (X, W)
theta np. random. rand (X. shape l1]) theta_yl np. random. rand (x. shape [1])
49
50
51
yo np. sum (X- theta_y0, 1) Y1 np. sum (X theta yl, 1)
54
np. arrarray [i] if w 0 else Yi[i] for i, w in enumerate (W) 1) + rp.random. randn (X. [0]) x.1
55
CATE Y1 YO
58
return YO, Y1, CATE,
[Step 6] Identifiability. All the above are simple functions, which are made identifiable (such that
they respect the DAG) through non-linearity. Then, X is generated by combining Z into X.
abs np.abs (X)
60
61
X_tilde - X. copy ()
tiiide :, dim: [__u====] = missinggvalii
tildde : dim] in==0] missing value
C ADDITIONAL EXPERIMENTS
Consider Table 3, where have repeated the experiment in Table 2 with GAIN [Yoon al., 2018]; 8]; a deep learning imputation method based on GANs [Goodfellow et al., 2014]. While notoriously difficult train, we find the combination of GAIN and MCM to an interesting one. Specifically, the discriminator is tasked with classifying imputed samples from real samples, making it harder to predict which samples were imputed. The way the discriminator does this, is by predicting binary mask of sorts, corresponding to what we have defined as Z. Having 50/50 prediction (resulting in the worst cross-entropy for binary prediction), we say that is independent of the imputed result, which then corresponds with theorem 1. Note that in theorem we pose independence of W. Seeing that *Z is mediator variable between W and "X, we have "X 1 W|X given that "X Z
As a second additional experiment, consider Fig. 4, where we have repeated the same experiment in Table 2, but with different levels of missingness. Not only do these results confirm that selective imputation is best across the board, they also bring to light an interesting phenomenon: the trade-off between prediction-gas from imputation versus bias reduction. As Y(w) is function of the fully
15

---

Under review as a conference paper at ICLR 2022
T-Learner
X-Leamer
1.2
0.8
PEHE
0.6
Nothing
0.8
0.4
Selective
0.6
Sel. Complement
0.4
0.2
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
amount of missingness
amount of missingness
Figure 4: Sensitivity analysis the amount of missingness: CATE. We report for two learners their performance (PEHE, y-axis) as function of the amount of missingness (x-axis). For each setting of the amount of missingness, sample 50 different train and test sets to calculate average and std (in line with what is reported in Table 2 and documented in our provided code-files).
observed X, it makes sense that predicting Y (w) from X directly would yield the best results. This insight is the motivation why imputation is so common in standard supervised learning. From our discussion in Section we learn that imputation in the treatment effects setting introduces bias, but perhaps there is only very little bias. In some cases, it might be better to impute, despite bias.
What we in Fig. is the combination of the balance trade-off described above, with another realisation: if too much data is missing, bias might actually reduce, as the associated patterns in the data become less apparent. Take the most extreme case where all data in "X is missing, Of course, then there is no bias anymore, as there is no more data to bias. With less bias in the data, it might be best for model to impute. This is clearly visible in the leftmost plot (T-Learner), where not imputing does not seem to improve with more missingness; only the "Sel. Complement" and "All"- which introduce bias less bias available to introduce by imputing. Overall, we find that "Selective" outperforms the rest.
to improve more data is missing, likely since there is
Table 3: CATE results with GAIN. We repeat our experiment form the main text. For each setting we report PEHE on 100 differently sampled simulations and averaged the results, standard devia- tion reported in brackets. Imputing only *X, while keeping X as is, consistently performs best across learners and treatments. Lower is better, our proposal is shaded
Impute
T-learner
Doubly rob.
X-learner
description
covariates
All
X* X
0.8720 (0.072)
1.4276 (1.410)
0.5388 (0.1
PEHE
Nothing
[0]
0.8233 (0.064)
0.2421 (0.579)
0.4724 (0.072)
Selective
[X]
0.5260 (0.105)
0.1497 (0.210)
0.2633 (0.069)
Sel. Complement
{X*}
0.8855 (0.056)
0.4726 (0.971)
0.6924 (0.152)
PEHEwa
All
(X+, X
0.8918 (0.107)
1.6743 (1.218)
0.5684 (0.138)
Nothing
[0]
0.7079 (0.087)
0.2719 (0.340)
0.4829 (0.116)
Selective
[XX]
0.5754 (0.092)
0.1539 (0.209)
0.2699 (0.064)
Sel. Complement
(X*)
0.8342 (0.105)
0.3531 (0.604)
0.7313 (0.200)
PEHEwa1
All
X+ X
0.8650 (0.074)
2,0708 (0.709)
0.5169 (0.107)
Nothing
(0)
0.8844 (0.080)
0.2798 (0.706)
0.4650 (0.070)
Selective
[X]
0.5035 (0.132)
0.1482 (0.213)
0.2590 (0.075)
Sel. Complement [X*}
0.9149 (0.068)
0.5332 (1.197)
0.6664 (0.
16

---

Under review as a conference paper at ICLR 2022
D EXHAUSTIVE DAG-SEARCH OVER CIT AND CIO
CIT and CIO are assumptions over X, X, Z, (ww), and W. While keeping discuss each other DAG that respects Eq. (2) or Eq. (3).
X
fixed,
In particular, for a DAG to be a valid description for missingness in CATE, they should contain: X which is a standard arrow in CATE (cfr. Fig. la). Essentially, the potential outcome (ww) is the result of natural process involving the fully observed X.
when a clinician determines treatment, they have to make due with what given to them, i.c. the observed covariate set X, which may be partially unobserved due to missingness. This proxy to X is the best a clinician has available to them as the fully observed X is not always available to them.
Z W throughout our work we consider Z to be different from X, where X entails the actual values of the covariate set, Z indicates their presence. If or not a variable is present for a clinician to base their treatment decision on, can have an effect on their eventual decision. Say that it is too risky for a particular treatment when a patient's blood-pressure is not observed, then a different treatment option will be chosen (or the variable will be measured before decision made). In this setting, the absence of value has determined treatment, leading Z into W.
Continuing our discussion, for a DAG to be valid as a missingness description, they should not contain the following:
X
W as, again, X is simply not available for clinician to base their treatment-decision on.
Y(w) is the result of natural process, it should depend on X, not X.
similarly, Y(w), should not depend the missingness Z different datasets on the same person would register different outcomes when outcomes depend directly Z, which cannot happen.
Permutations CIT. For CIT, there cannot be direct arrow from X W (a feat agree with), as it automatically violates Eq. (2). As long as there exists path between W and Y, excluding the connecting SWIG-path [Richardson & Robins, 2013], we consider the DAG valid. When excluding
paths going from W or (w)) the total amount of valid DAGs amounts to (C' +C3 CG3) (C)) + C3) = 21, where O3 is coming from three potential paths into Y (having three variables, other than W and Y (w)), and C2 is coming from two potential paths into W (three variables, excluding X). All the CIT DAGs, including the one presented in Mayer et [2020b] are illustrated in Fig. (the DAGs presented in [Mayer et al., 2020b] correspond to Figs. 5a and 6a for CIT and CIO, respectively).
Figure organised as follows: each two rows contain 7 graphs (corresponding to (C} + C3 + ())) where we vary the arrows going into Yww); there are three sets of graphs (where set is two rows), for each set we vary the arrows going into W (where three then corresponds to (C) + C2))..
Handy with Table 4, we find that Fig. 5g is the only valid DAG from all CIT (and later see all CIO) compatible DAGs. Inspecting Fig. 5g we notice that it corresponds exactly with Fig. 1d, without the birdirectional arrow between W and Z, which is result of informative versus uninformative missingness. Splitting Z in Z+ and Z results automatically in MCM.
Permutations CIO. For CIO, the definition does not allow an arrow from X to Y(w), similarly to the definition of CIT not allowing an arrow from X to W. Contrasting CIT, however, we find this restriction not sensible. As we have already argued above, X is the only variable "justified" to influence the potential outcomes, Y(w), directly. Any other variable influencing Y(w) would result in contrasting outcomes over different datasets, which would imply, for example, different tumour sizes for the same person across different datasets. Having CIO and accompanying Eq. (3), results in 21 (applying the same calculation as for CIT) non nen-sensible DAGs, including the DAG presented in Mayer et al. [2020b]. All these DAGs are listed in Fig. 6, and evaluated (like for CIT) in Table 4.
17

---

Under review as a conference paper at ICLR 2022
Table 4: Validity of CIT and CIO. There are six criteria we argue missingness in CATE should follow. These criteria are indicated in the column headers as directed dependencies missingness should, or should not, include. Assuming that there are no arrows going up from W and Y Y(w) (for example, the potential outcomes (w)), cannot influence the covariates X), we have 42 DAGs that respect Eq. (I) and one of Eq. (2) or Eq. (3). Only one of these DAGs (indicated in green ), respects each criteria. This DAG (Fig. 5g) is permutation of CIT, and version of MCM that does not assume factors, Z and z2 in Z. In below table, "V" means presence and "X" means absence, when these icons black absence presence is positive, when they are gray they negative.
Does not contain
Does contain
W
Fig. 5a
assum.
Fig. 5b
assum.
Fig. 5c
A
Fig. 5d
assum.
Fig. 5e
Fig. 5f
assum.
Fig. 5g
assum.
Fig. 5h
assum,
Fig.
assum.
Fig. 5j
assum.
Fig. 5k
assum.
Fig. 51
assum.
Fig. 5m
assum.
Fig. 5n
assum
Fig. 5o
assum.
5ig.5p Fig. 5q
assum.
Fig. 5r
assum.
Fig. 5s
assum.
Fig. 5t
assum.
Fig. 5u
assum.
Fig. 6a
Fig. 6b
Fig. 6c
Fig. 6d
Fig. 6e
Fig. 6f
Fig. 6g
Fig. 6h
Fig.
Fig. 6j
C10
Fig. 6k
Fig. 61
Fig. 6m
Fig. 6n
Fig. 60
Fig. 6p
Fig. 6q
Fig. 6r
Fig. 6s
Fig. 6t
Fig. 6u
18

---

Under review as a conference paper at ICLR 2022
W
W
W
Y
W
(a) Original CIT
(b)
(c)
(d)
X
Z
X
Z
X
Z
W
Y
W
W
(e)
(g)
X
W
W
Y
W
Y
(h)
ii)
(k)
X
Z
X
Z
X
Z
W
W
W
(
(m)
X
X
X
X
X
X
W
Y
W
W
(0)
(p)
(q)
(n)
X
Z
X
Z
X
Z
W
W
W
(u)
Figure 5: Permutations on CIT. There are a total of DAGs that respect Eq. (1) and Eq. (2). From these DAGs, only one Figure 5g is acceptable. Add only the assumption that some elements in Z are informative, and some elements in Z are uninformative, and we automatically arrive at MCM. Note that we exclude edges going up from W Y (w), similarly Richardson & Robins [2013].
19

---

Under review as a conference paper at ICLR 2022
X
X
W
W
W
(a) Original CIO
(b)
(c)
(d)
X
7
X
7
X
Z
X
W
W
W
X
X
X
X
Y
W
y
(h)
(k)
W
W
W
(I)
(m)
(n)
X
x
X
X
X
X
X
Y
W
Y
W
Y
W
Y
(0)
(p)
(g)
(r)
X
Z
X
Z
X
Z
W
W
W
(u)
Figure 6: Permutations CIO. There 21 possible DAGs that respect Eq. (3). None of them
is acceptable as they, by definition, cannot include direct edge from X to Y (w). Having direct edge between X and (w) encodes dependence, despite conditioning X and Z. Note that have excluded edges going up from or Y (w), in similar fashion to Richardson & Robins [2013].
20


Reviews:
[{"review_id": "vop8d7q0nC4", "reviewer": "Reviewer_efJq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths of paper**\n\n- Paper presents a very elaborate writeup with a detailed introduction to causality and missingness in treatment effect estimation which makes it a good read for audience who are new to this field.\n\n- Problem addressed is a relevant problem and is applicable across several real-world problems.\n\n**Weakness**\n\n- The paper has a lot of content on introducing causality which instead could have been used to add more use cases on how selective imputation can be useful for TE\n- It does appear that the authors could have studied the exhaustiveness claim of MCM in depth to tease out the contributions of this paper better. This is a very broad claim with insufficient justification in the current draft.\n- It is not exactly clear how this paper advances the technical state-of-the-art to clear the ICLR bar of acceptance. The general writeup and contributions make it a better fit for a venue like UAI and other similar avenues.\n\nOverall, I would like the authors to think more broadly on the imputation problem itself as industry problems rarely rely on using any kind of imputation method (mainly due to skepticism and avoiding corrupting the data). It might help if authors can talk about broader adoption for this work for such a real-world setting.", "summary_of_the_paper": "In this paper, the authors mainly study the problem of missing data in treatment effect estimation and highlight the importance of addressing this problem. The authors propose a selective imputation scheme which is more well suited for addressing missingness in such scenarios. Authors also present several sample scenarios illustratively which indicate potential issues with current methods while doing TE with missingness.  Empirical results compare different scenarios where general imputation schemes (imputing all data, no data, wrong data) can be much worse than their proposed method.", "summary_of_the_review": "I have highlighted my major observations from this paper above. While the paper is written in a very elaborate manner, I do believe it is still not clear if the technical contributions are advancing the state-of-the-art to clear the ICLR bar of acceptance. Authors should tease out the technical contributions around exhaustiveness and others more clearly and remove big sections on assumptions, metrics (can be written briefly with citations). I would also recommend trimming down the graphs to only keep the most relevant DAGs with a clear messaging.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Sw15Q7x-eNY", "reviewer": "Reviewer_8bft", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Authors use graphic causal models to model the various missingness in causal effect estimation. This is an interesting exercise.\n\nHowever, I doubt the correctness of mapping Figure 1(d) to Figure 3. In Figure 1(d) the relationships between the rightmost four variables are \"W \\to Z_2 \\to X'_2 \\leftarrow X\" (Note that I use Z_2 and X'_2 to replace the superscripted Z and X in Figure 1(d)). We should understand the relationships in the following way. W determines the missing values of Z_2 and subsequently the observed values of X'_2. This does not mean that W will determine X_2 when it does not include missing values. Using the authors' example, the participants of the job training program provide additional information whereas the non-participants do not. So, the missing values in X'_2 are determined by W. However, if we have a magical way to collect the additional information from non-participants (i.e. the missing values are imputed perfectly). The edge W \\to real X_2 will be broken. Figure 1(d) becomes Figure 1(c). If X_2 is imputed correctly, there is not an edge W \\to X_2 and the discission before \"selective imputation\" on Page 7 is invalid.   \n\nGiven that Figure 3 is confusing and leads to an incorrect conclusion. There is a soundness problem with the paper.\n\nSince the experiments follow Figure 3, the results cannot be trusted. If X_2 is a part of the covariate set, there is no edge W \\to X_2 without missing values. The causal effect on the set of X is unbiased. If X_2 is imputed correctly, the causal effect is also unbiased. \n\nInterestingly, there is not a discussion of how the missing values are imputed.\n    ", "summary_of_the_paper": "This paper studies dealing with missing values in estimating treatment effects. Authors identify a new missingness mechanism, mixed confounded missingness (MCM), including missingness that determines treatment selection and missingness that is determined by treatment selection. The authors show that both imputation and no imputation lead to poor treatment effect estimations. The authors present a selective imputation strategy that informs which variables should be imputed and which should not. They empirically demonstrate the effectiveness of the strategy.\n", "summary_of_the_review": "There is a soundness problem with the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "-aEdXoBehI4", "reviewer": "Reviewer_ZWmF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The questions they are trying to solve is an important problem.\nFor example in the discussion the author emphasize that \"more care and thought should\nbe put into imputing missing data when estimating treatment effects.\" which I agree with completely.\n\nThe authors are using simplified DAG graphic tool to modeling the exposure, mediator, outcome variables with missingness and propose the causal relationship of missingness. In the real world data, one covariate may cause missingness of an exposure, which could be the predictor of the outcome variable. However, the missingness of this covariate may have no direct correlation with outcome variable. Therefore, to determine which variable will be imputed or not will not be simply determined by strategy will be limited.\n\nMissingness in one variable may have direct or indirect relationship with missingness in the other variable or outcome variable. Alternative speaking, this correlation may not be linear. Which variables will be imputed or not imputed for the prediction of outcome variable will be  determined using machine learning approach for the feature selection.\n\nFinally, even though it is difficult to use real word data for experimentation, it is important to attempt to do that. The imputation procedure has to be designed for a specific field as pattern of missingness may be very different among disciplines. The pattern of missingness and mechanism for a given dataset should be recognized. Whether the competition is favoring a certain method or procedure has to be determined in the “real-world” data with “real-world” missingness by considering recognized and unrecognized missing pattern/mechanism, as well as the plausible distribution of missing data. Testing a method on synthetic data, with no regards to observed patterns of missingness may only add noise to the field. A recent paper in the field was just published this month: https://www.nature.com/articles/s41746-021-00518-0 \n\nTherefore, even though the idea of improving imputation is excellent, the logic used in this study is not robust.  ", "summary_of_the_paper": "\"we identify a new missingness mechanism, which we term mixed confounded missingness (MCM), where some missingness determines treatment selection and other missingness is determined by treatment selection.\"\n\nThe author gave a new term called “MCM” which is not something new. MCM is a type of MNAR (Missing-Not-At-Random). \nMCM cannot be listed parallel to MCAR, MAR and MNAR. \n\n", "summary_of_the_review": "Even though the idea of improving imputation is excellent, the logic used in this study is not robust and does not take into account true realistic situations.  \n\nUse of only synthetic data, when we have access to many real-world datasets is not acceptable. How the synthetic data was created has a direct effect on how the methods will work. It is possible to use real-world data and introduce additional hold-out values to evaluate the methods. see for example: https://www.nature.com/articles/s41746-021-00518-0\n\nThe importance of real-world data is key here because the whole idea behind this paper is that missingness patterns are not completely understood. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors propose a new type of (missing not at random) model they call the MCM (mixed confounded missingness).
The authors further discuss that given their model, naive imputation strategies do not work, and a model-tailored imputation strategy is needed.

The reviewers did not receive the paper favorably, with main complaints centering around: (a) outlining novelty compared to existing approaches to missing data, (b) whether imputation is a good strategy for dealing with missing data, and (c) whether the paper's results are actually sound.

Here's my perspective on these worries.

The paper aims to deal with missing data in a causal inference context (in other words, the target of inference is a causal effect, and our data happens to have entries missing not at random).  Further, the paper aims to work within a graphical modeling formalism for missing data models.  Finally, the paper points out that imputation is to be done with care if data is missing not at random (a point both myself, and reviewers agreed with).

Areas of improvement in the paper, in my mind, would be: (i) better literature review and putting authors' work in context of prior work, (ii) being clear about identification, and (iii) discussion of estimation strategies (not just imputation).

Dealing with missing data (in particular right censoring, but also more general types of missingness) in causal inference is a very old problem, with an established literature in statistics and public health.  In fact, methods for dealing with both causal inference and missing data together are a part of standard graduate curriculum in epidemiology and biostatistics in many Universities.

(i) Literature review and context.  Some papers the authors may find helpful to review:

James M. Robins, Andrea Rotnitzky, Daniel O. Scharfstein.  Sensitivity Analysis for Selection bias and unmeasured Confounding in missing Data and Causal inference models.  Part of the The IMA Volumes in Mathematics and its Applications book series (IMA, volume 116).

This paper discusses lots of relevant things, but in particular sensitivity analysis methods to violations of MAR in settings the authors worry about.

James M. Robins. Non-response models for the analysis of non-monotone non-ignorable missing data. Statistics in Medicine, 16:21–37, 1997.

This paper is an early example of an MNAR model that may be represented by a directed acyclic graph.

Karthika Mohan, Judea Pearl, and Jin Tian. Graphical models for inference with missing data. In C.J.C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 1277–1285. Curran Associates, Inc., 2013.

Ilya Shpitser, Karthika Mohan, Judea Pearl.  Missing data as a causal and probabilistic problem.  In Proceedings of the Thirty First Conference on Uncertainty in Artificial Intelligence (UAI-15), pp. 802-811, AUAI Press, 2015.

Rohit Bhattacharya, Razieh Nabi and Ilya Shpitser. “Full Law Identification In Graphical Models Of Missing Data: Completeness Results.” In Proceedings of the Thirty-Seventh International Conference on Machine Learning (ICML-20), pp. 7153-7163, 2020.

These papers deal with general models of missing data using graphs.

Since the authors use graphical models as well, I urge them to put their contribution in context with this prior work.

(ii) Identification.  The authors should clearly discuss whether treatment effects are identified under their model, and if so, by what function.  If this function is not closed form (which can happen in missing data), this should be discussed as well.  This should be contrasted with other missing data work that derives identification under MNAR, particularly using graphs.

(iii) Estimation.  The authors chose to use imputation.  Imputation is a sampling approach to inference in missing data.  Others include maximum likelihood or Bayesian methods (via EM), or semi-parametric inference via influence functions.  If the authors chose to concentrate on imputation, specifically, they should explain why (as other methods have noted advantaged, e.g. statistical efficiency, quantification of uncertainty, etc.).

Cautioning against naive imputation is a fine thing to do, but everyone working on missing data problems already knows naive imputation does not work for MNAR data.  Please do not oversell your contributions.  Saying things like: "MCM being the first formalisation of a missingness mechanism when there are treatments at play." is neither true, nor helpful for the peer review process.

With all that said, the MCM model has the potential to be an interesting MNAR model, and placed in proper context of existing work, could be a very interesting addition to the missing data literature.  However, the draft needs a bit more work before it is ready for publication.
Decision: Reject