MIRACLE: Causally-Aware Imputation via Learning Missing Data Mechanisms
Year: 2022

Abstract:
Missing data is an important problem in machine learning practice. Starting from the premise that imputation methods should preserve the causal structure of the data, we develop a regularization scheme that encourages any baseline imputation method to be causally consistent with the underlying data generating mechanism. Our proposal is a causally-aware imputation algorithm (MIRACLE). MIRACLE iteratively refines the imputation of a baseline by simultaneously modeling the missingness generating mechanism, encouraging imputation to be consistent with the causal structure of the data. We conduct extensive experiments on synthetic and a variety of publicly available datasets to show that MIRACLE is able to consistently improve imputation over a variety of benchmark methods across all three missingness scenarios: at random, completely at random, and not at random.

heading: 1 Introduction
text: Missing data is an unavoidable byproduct of collecting data in most practical domains. In medicine, for example, doctors may choose to omit what they deem to be irrelevant information (e.g., some patients may be asked to get comprehensive blood tests while others don’t), data may be explicitly omitted by the patient (e.g., avoiding questions on smoking status precisely because of their smoking habit) or simply misrecorded in electronic health systems (see e.g., [3, 12, 28]). Imputation algorithms can be used to estimate missing values based on data that was recorded, but their correctness depends on the type of missingness. For instance, expanding on the example above, younger patients may also be more likely to omit their smoking status. As illustrated in Figure 1, the challenge is that
implicitly conditioning inference on observed data introduces a spurious path of correlation between age and the prevalence of smoking that wouldn’t exist with complete data.
Figure 1: Missingness may introduce spurious dependencies.
Missing data creates a shift between the available missing data distribution and the target complete data distribution. It is a shift that may be explicitly modeled as missingness indicators in an underlying causal model (i.e., a missingness graph as proposed by Mohan et al. [19]) as shown in Figure 1. The learning problem is one of extrapolation, learning with access to a missing data distribution for
⇤Equal contribution
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
prediction and inference on the complete data distribution – that is, generated from a model where all missingness indicators have been intervened on (interventions interpreted in the sense of Pearl [22]) thus graphically removing the dependence between missingness and its causes, and any spurious correlations among its ancestors.
With this causal interpretation, imputation of missing data on a given variable Y from other observed variables X is formulated as a problem of robust optimization,
minimize ✓2⇥ sup P2P
E(X,Y )⇠P ⇥ (f✓(X) Y )2 ⇤ , (1)
simultaneously optimizing over the set of distributions P arising from interventions on missingness indicators. Causal solutions – i.e. imputation using functions of causal parents of each missing variable in the underlying causal graph – are a closely-related version of this problem with an uncertainty set Q defined as any distribution arising from interventions on observed variables and variable indicators (see e.g. sections 3.2 and 3.3 in [16]),
sup P2P
E(X,Y )⇠P ⇥ (f✓(X) Y )2 ⇤  sup
P2Q E(X,Y )⇠P
⇥ (f✓(X) Y )2 ⇤ , (2)
since P ⇢ Q. Our premise is that causal solutions, i.e. minimizing the right-hand-side of (2), are expected to correct for spurious correlations introduced by distribution shift due to missing data and preserve the dependencies of the complete data for downstream analysis.

heading: 1.1 Contributions
text: In this paper, we propose to impute while preserving the causal structure of the data. Missing values in a given variable are replaced with their conditional expectation given the realization of its causal parents instead of the more common conditional expectation given all other observed variables, which absorbs spurious correlations.
We propose a novel imputation method called Missing data Imputation Refinement And Causal LEarning (MIRACLE). MIRACLE is a general framework for imputation that operates on any baseline (existing) imputation method. A visual description is given in Figure 2: given some initial imputation from a baseline method, MIRACLE refines its imputations iteratively by learning a missingness graph (m-graph) [19] and regularizing the imputation function such that it is consistent with the causal graph generating the data, substantially improving performance. In experiments, we apply MIRACLE to improve six popular imputation methods as baselines. We present detailed simulations to demonstrate on synthetic and a variety of publicly available datasets from the UCI Machine Learning Repository [8] that MIRACLE can improve imputation in almost every scenario and never degrades performance across all imputation methods.

heading: 1.2 Related work
text: The literature on imputation is large and varied. Still, most imputation algorithms work with the prior assumption that the missing data mechanism is ignorable, in the sense that imputation does
not require explicitly modeling the distribution of missing values for imputation [25]. Accordingly, classical imputation methods impute using a joint distribution over the incomplete data that is either explicit or implicitly defined through a set of conditional distributions. For example, explicit joint modeling methods include generative methods based on Generative Adversarial Networks [33, 34, 1], matrix completion methods [15], and parametric models for the joint distribution of the data. Missing values are then imputed by drawing from their predictive distribution. The conditional modeling approach [31] consists of specifying one model for each variable and iteratively imputing using estimated conditional distributions. Examples of discriminative methods are random forests [27], autoencoders [11, 10, 14], graph neural networks [35], distribution matching via optimal transport [20], and multiple imputation using chained equations [6].
In a different line of research, Mohan et al., in a series of papers, see e.g. [19, 18], explicitly considered missing data within the underlying causal mechanisms of the data. Subsequently, a range of related problems has been studied, including identifiability of distributions and causal effects in the presence of missing data, see e.g. [5, 26, 21], testable implications relating to the causal structure using missing data [18], and causal discovery in the presence of missing data [9, 30]. Our focus, in contrast, is algorithmic in nature. We aim to develop an algorithm that improves imputation quality by leveraging causal insights represented as an estimated missingness graph learned from data.

heading: 2 Background
text: The basic semantic framework of our analysis rests on structural causal models (SCMs) (see e.g. Chapter 7 in [22] for more details) explicitly introducing missingness indicators and their functional relationship with other variables, using in part the notation of [19]. We define an SCM M as a tuple (X,R,U ,F , P ) where X is a vector of d endogenous variables and U is a vector of exogenous variables.2 R is the vector of missingness indicators that represent the status of missingness of the endogenous variables X . Precisely, Rj is responsible for the value of a proxy variable Zj of Xj , i.e., the observed version of Xj . For example, Zj is equal to Xj if the corresponding record is observed (Rj = 1), otherwise Zj is missing (Rj = 0). F is a set of functions where each fX , fR 2 F decide the values of an endogenous variable X and a missingness indicator variable R, respectively. The function fX takes two separate arguments as parts of X (except X itself) and U , termed as PaX and UX . That is, X fX(PaX , UX) and R fR(PaR, UR). The randomness in SCMs comes from the exogenous distribution PU (u) where the exogenous variables in U are generated independently and are mutually independent. Naturally, through the functions in F , the SCM M induces a joint distribution PX(x) over the endogenous variables X , called the endogenous distribution. An intervention on some arbitrary random variables V in X and R, denoted by do(v), is an operation which sets the value of V to be v, regardless of how they are ordinarily determined. For an SCM M, let Mv denote a submodel of M induced by intervention do(v). The interventional distribution PX(x|do(v)) induced by do(v) is defined as the distribution over X in the submodel Mv , namely, PX,Mv (x) = PX(x|do(v)). Each SCM in the context of missingness is associated with a m-graph G (e.g., Fig. 1a), which is a directed acyclic graph (DAG) where nodes represent endogenous variables X and missingness indicators R, and arrows represent the arguments PaX and PaR of each function fX and fR respectively. By convention, exogenous variables U are often not shown explicitly in the graph.
Assumption 1 (Missingness indicators are not causes) No missingness indicator in R can be the cause of the endogenous variables X , i.e., the arguments of the functions generating X .
Assumption 2 (Causal sufficiency) Exogeneous variables U are mutually independent, i.e., all common parents of the endogenous variables are included in X .
Assumption 3 (No self-masking missingness) Self-masking missingness refers to missingness in a variable that is caused by itself. In the m-graph this is depicted by an edge from Xj to Rj (as shown in Figure 3 (d)). We assume that there is no such edges in the m-graph.
Assumption 4 (Observed root nodes) The endogenous variables Xj such that PaXj = ; (i.e., the root nodes) in the m-graph are always observed (Rj = 1 with probability 1).
2Essentially, X is the ground-truth features; U is the random noise in the data generating process.
We make the four assumptions above throughout the following sections. Assumption 1 and 2 are employed in most related works using m-graphs (see e.g. [18, 19]). Assumption 1 is valid, for example, if R is generated in the data collection process after the variable values are assigned. Consequently, under this assumption, if two endogenous variables of interest X1 and X2 are not d-separated by some variable X3, they are not d-separated by X3 together with their missingness indicators R1 and R2. We denote an independent relation in a data distribution by “ |= ” and dseparation in a m-graph by “ |= d”. We assume the data distribution is faithful to a m-graph, meaning that the two independencies are equivalent. As shown in Figure 3, data is missing completely at random (MCAR) if X |= dR holds in the m-graph, missing at random (MAR) if for any endogenous variable Xj , Rj |= dXj | X j holds, and missing not at random (MNAR) otherwise, as stated in [19]. If Assumption 3 is violated, we are unable to learn the missingness for self-masked variables. Assumption 4 is necessary for imputing all the missing variables from their causal parents. These assumptions are imperative for MIRACLE to provide improved imputations by leveraging the causal structure of the underlying data generating process. In our experiments (Section 4), we apply MIRACLE to real-world datasets where these assumptions are not guaranteed.

heading: 2.1 Why is imputation prone to bias?
text: The reason for considering the causal structure of the underlying system is that when learning an imputation model from observed data, implicitly conditioning on some missingness indicators in R induces spurious dependencies that would not otherwise exist. For example, in a graph X1 ! R3 X2, conditioning on R3 = 1 induces a dependence between X1 and X2. In general, the distributions PX(x|R = r) and PX(x|do(r)) differ unless missingness occurs completely at random, and motivates an interpretation of the problem as domain generalization, training on data from one distribution ultimately to be applied on data from a different distribution that, in our case, arises from missing data (i.e., interventions on missingness indicators). This shift is not addressed in the imputation methods that only use the feature correlations.

heading: 3 MIRACLE
text: In this section, we propose to correct for the shift in distribution due to missing data by searching for causal solutions and explicitly refining imputation methods using a penalty on the induced causal structure. In practice, we have n i.i.d. realizations of the observed version of X 2 Rd, concatenated as a incomplete data matrix X 2 Rn⇥d, together with missingness indicators concatenated in a matrix R 2 {0, 1}n⇥d. We use here the same bold uppercase notation for sets of variables and matrices of observations but their meaning should be clear from the context. Our goal is to impute the unobserved values in X using each variable’s causal parents. We define the imputed data X̃ 2 Rn⇥d,
X̃ = R X+ (1 R) X̂
where is the element-wise product of matrices and X̂ is an estimate of the complete data matrix.

heading: 3.1 Network architecture
text: In this section, we describe our approach for estimating X̂. Let dS  d be the number of partially observed features, i.e., missing for at least one realization. S is the set of missing features indices. The imputation network is defined as a function f : Rd ! Rd ⇥ [0, 1]dS that takes an initially imputed dataset X̃(0) (using an existing baseline imputation method), and returns two quantities:
1. A refined imputation X̂.
2. An estimation of the probabilities of features Xij being missing, i = 1..., n and j 2 S .
A depiction of the network architecture and optimization algorithm is shown in Figure 4. The architecture is constructed with respect to the assumptions shown in Section 2. Our model f is decomposed into two sub-networks, f = (f (imp), f (miss)), responsible for imputing the unobserved data and estimate the probabilities of missingness, respectively. The imputation network has d components, f (imp) = (f (imp)1 , . . . , f (imp) d ), one for each variable, and the missingness network has dS components, f (miss) = (f (miss) 1 , . . . , f (miss) dS
). Each component, for both networks, has separate input and output layers but shared hidden layers (of size h). Let W(imp)1,j and W (miss) 1,j denote the h⇥ d weight matrix (we omit biases for clarity) in the input layer of f (imp)j and f (miss) j respectively. The j-th column of W(imp)1,j and W (miss) 1,j is set to 0. Let Wm 2 Rh⇥h, for m = 2, . . . ,M 1, denote the weight matrix of each hidden layer and let W(imp)M,j and W (miss) M,j , be the 1⇥h dimensional output layers of each sub-network. The imputation network prediction is given by,
f (imp)j (x) := W (imp) M,j (· · · (W2 (W (imp) 1,j x))),
for j = 1, . . . , d. And similarly, the missingness network prediction is given by,
f (miss)j (x) := W(miss)M,j (· · · (W2 (W (miss) 1,j x))) ,
for j = 1, . . . , dS , where (·) is the ELU activation function and is the sigmoid function. Our network is optimized with respect to three objectives. First, to accurately predict missing values, second, to faithfully encode the causal relationships given by the underlying m-graph, and third to satisfy a moment constraint of the missing data mechanism on the imputed values.

heading: 3.2 Reconstruction loss
text: The first objective is to train f to correctly reconstruct each feature from the observed data using a reconstruction loss,
L1 = 1
n
nX
i=1
xi ri f (imp)(x̃(0)i ) ri 2 +
nX
i=1
CrossEntropy h r̃i, f (miss)(x̃(0)i ) i! ,
where xi and x̃ (0) i are the realized and imputed feature vector of the i-th instance, r̃i are the dS components of ri that are missing for at least one instance. The first loss term is for reconstructing the observed features, and the second loss term is for estimating the probabilities of missingness.

heading: 3.3 Causal regularizer
text: The second objective is to ensure that the dependencies defined by f correspond to a DAG over the features X and the missing indicators RS , which enforces that the learned functional dependencies recover a DAG in the equivalence class of causal graphs over the observed data. Enforcing the acyclicity of the dependencies induced by a continuous function f is originally proposed in [36, 37]. Define a binary adjacency matrix B 2 {0, 1}(d+dS)⇥(d+dS); [B]k,j = 0 (i.e., the l2-norm of the k-th column of the matrix W(imp)1,j or W (miss) 1,j is 0) is a realistic and sufficient condition for achieving @kfj = 0. The adjacency matrix B of the graph induced by the learned f is acyclic if and only if,
R1 = 12h 2(B) + h(B), (3)
is equal to zero, where h(B) := Tr(exp{B B}) (d+ dS) and exp(·) is the matrix exponential. Remark 1. Existing imputation methods based on feature correlations essentially assume an undirected (non-causal) graph between the features. Further, acyclicity is a realistic and practical assumption to make on the static datasets collected by human experts. In nature, most data distributions generate their features in some order. In a directed graph, a cycle means a path starts and ends at the same node. This is unlikely to happen in the data generating process if not considering variables over time, i.e., time-series data. By enforcing acyclicity, MIRACLE only uses the causal parents for imputation, which is less biased by spurious dependencies that only exist in the observed data.

heading: 3.4 Moment regularizer
text: The third objective leverages a set of moment constraints in the missingness pattern to improve imputation. Assume ⇠j = P Rj = 1 | PaRj ) 2 ( , 1 ), for some > 0. The following derivation holds for MAR or MCAR missingness patterns only. It holds that,
E ⇢ RjXj ⇠j = E ⇢ E  RjXj ⇠j Xj , PaRj
= E ⇢ Xj ⇠j E ⇥ Rj | Xj , PaRj ⇤ = E {Xj} , (4)
where the third equality follows from the MAR assumption (Rj ? Xj | X j). Under the MCAR assumption, this derivation holds trivially since in that case Rj ? Xj . We can use the missingness and imputation networks to enforce the above equality algorithmically, ensuring the left hand side equals the right hand side in the empirical version of (4) as follows,
R2 = dSX
j=1
[⌧̂j,SIPW ⌧̂j,mean]2 = dSX
j=1
2
4
nX
i=1
eijrij
! 1 nX
i=1
eijrijxij 1
n
nX
i=1
f (imp)S[j] (x̃ (0) i )
3 5 2
,
where eij = 1/f (miss) j (x̃ (0) i ), and S[j] is the j-th element of S, i.e., the index of the j-th missing feature. Minimizing R2 forces the two estimators of E {Xj} to match, the stabilized inverse propensity score weighting (SIPW) estimator ⌧̂j,SIPW [24] using the missingness network f (miss) j (in eij) and the mean estimator ⌧̂j,mean using the imputation network f (imp) S[j] .
Remark 2. We hypothesize this mechanism can improve performance for two reasons. First, the missing data mechanism P (Rj = 1 | PaRj ) can be a simpler function that takes less samples to learn than the function that generates the feature j, E ⇥ Xj |PaXj ⇤ . Then the SIPW estimator based on f (miss)j will converge to the true mean faster than the estimator based on f (imp) S[j] . Second, in R2, the mean estimator using f (imp)S[j] is based on all the samples; f (imp) S[j] is trained to produce predictions on the samples with missing feature j for the sake of matching the SIPW estimator. By contrast, without the regularizer R2, f (imp)S[j] is solely trained on the samples with observed feature j, and its performance may fail to generalize to data with missing feature j.

heading: 3.5 Bootstrap Imputation
text: Discovering a causal graph requires complete data. However, this is not the case for missing data problems. Because of this, we require that MIRACLE be seeded by another imputation method. Imputed values are iteratively refined by MIRACLE, hence “bootstrapping”, to potentially converge to a new imputation that minimizes MIRACLE’s objective (including causal and moment regularizers). MIRACLE’s objective for optimization is,
L = L1 + 1R1 + 2R2, (5)
where 1 and 2 are hyperparameters that define the strength of regularization. We iteratively update the baseline matrix X̃(0) with a new imputed matrix X̃ given by MIRACLE every ten epochs in training. With increasing epochs, stochastic optimization minimizes the loss for the imputed matrices that respect the causal and moment regularization. In theory, this is analogous to supervised training a denoising autoencoder (DAE) [32, 4, 23], but differs only by the fact that “noise” comes from prior or previous imputations. In training DAE, the input samples are corrupted by independent noise
with each epoch, yet convergence is still guaranteed [2]. In our experiments, we demonstrate that bootstrap imputation indeed converges on multiple datasets and baseline methods. In Appendix E, we provide experiments on the computational complexity of MIRACLE.

heading: 4 Experiments
text: In this section, we validate the performance of MIRACLE using both synthetic and a variety of real-world datasets.
1. In the first set of experiments, we quantitatively analyze MIRACLE on synthetic data generated from a known causal structure.
2. In the second set of experiments, we quantitatively evaluate the imputation performance of MIRACLE using various publicly available UCI datasets [8].
General set-up. We conduct each experiment five times under random instantiations of missingness. We report the RMSE along with standard deviation across each of the five experiments. Unless otherwise specified, missingness is applied at a rate of 30% per feature. For MCAR, this is applied uniformly across all features. For MAR, we randomly select 30% of the features to have missingness caused by another disjoint and randomly chosen set of features. Similarly, we randomly select 30% of features to be MNAR. We induce MAR and MNAR missingness using the methods outlined in [33], and we provide more details in Appendix A.2.
We use an 80-20 train-test split. We performed a hyperparameter sweep (log-based) for 1 and 2 with ranges between 1e-3 and 100. By default we have 1 and 2 set to 0.1 and 1, respectively.
Evaluating imputation. For each subsection below, we present three model evaluations in terms of missingness imputation performance, label prediction performance of a prediction algorithm trained on imputed data and the congeniality of imputation models.
• Missingness imputation performance is evaluated with the root mean squared error comparing the imputed missing values with their actual unobserved values.
• Label prediction performance of an imputation model is its ability to improve the postimputation prediction. By post-imputation, we refer to using the imputed data to perform a
downstream prediction task. To be fair to all benchmark methods, we use the same model (support vector regression) in all cases.
• The congeniality of an imputation model is its ability to impute values that respect the feature-label relationship post imputation. Specifically, we compare, support vector parameters, w, learned from the complete dataset with the parameters ŵ, learned from the imputed dataset. We report root mean square error (||w ŵ||2)1/2 for each method. Lower values imply better congeniality [33].
Baseline imputation methods. We apply MIRACLE imputation over a variety of six commonly used imputation baseline methods: (1) mean imputation using the feature-wise mean, (2) a deep generative adversarial network for imputation using GAIN [33] (3) k-nearest neighbors (KNN) [29] using the Euclidean distance as a distance metric of each missing sample to observed samples, (4) a tree-based algorithm using MissForest (MF) [27], (5) a deep neural distribution matching method based on optimal transport (OT) [20], and (6) Multivariate Imputation by Chained Equations (MICE) [6]. For each of the baseline imputation methods with tunable hyperparameters, we used the published values. We implement MIRACLE using the tensorflow3 library. Complete training details and hyperparameters are provided in Appendix D.
Additional experiments. In Appendix B and C, we also evaluate MIRACLE in terms of providing imputations that improve predictive performance [33, 35], as well as, in its ability to impute values that respect the feature-label relationship, i.e., congeniality [17, 7]. We also investigate the impact of the graphical missingness location on imputation performance on synthetic data in Appendix G.

heading: 4.1 Synthetic data
text: In this subsection, we evaluate MIRACLE on synthetic data. In doing so, we can control aspects of the underlying data generating process and possess oracle knowledge of the DAG structure.
Data generating process. We generate random Erdos-Renyi graphs with functional relationships from parent to children nodes. At each node, we add Gaussian noise with mean 0 and variance 1. For a complete description of the underlying data generating process, see Appendix A.
Synthetic results. In Figure 5, we show experiments of MIRACLE on synthetic MAR data in terms of RMSE. Our experiments show that MIRACLE is able to significantly improve imputation over each of the baseline imputation methods. Figure 5 shows MIRACLE improves performance over each baseline method across various dataset sizes, missingness ratios, and feature sizes (DAG sizes). We show results for MCAR in Appendix B.

heading: 4.2 Experiments on real data
text: In Figure 6 we show experiments of MIRACLE on real data. We perform experiments on several UCI datasets used in [33, 35, 20, 13]: Autism, Life expectancy, Energy, Abalone, Protein Structure, Communities and Crime, Yeast, Mammographic Masses, Wine Quality, and Facebook Metrics. In Figure 6, the improvements of MIRACLE are minimal for MCAR (except for mean imputation). This agrees with our discussion in Section 2.1, because the baseline imputations are not biased in the MCAR setting where X |= dR holds in the m-graph. Conversely for the MAR and MNAR settings, as expected, we observe MIRACLE has an significant improvement on some of the datasets, such as Abalone, Autism, Energy and Protein Structure. As discussed in Section 2, MIRACLE can improve the baseline imputation under Assumptions 1-4, which may not hold in these real-world datasets. Nevertheless, we observe that MIRACLE never degrades performance relative to its baseline imputation on any dataset. Furthermore, no baseline imputer is optimal across the datasets. In almost all cases, applying MIRACLE to any baseline results in the lowest error. We show a similar gain for this experiment with MCAR and MNAR in Appendix C. An ablation study on our overall loss is provided in Appendix F.
3Source code at https://github.com/vanderschaarlab/MIRACLE.

heading: 4.3 Causal discovery and imputation performance
text: In our experiments, we observe a positive correlation between the quality of learned DAGs (and causal parents) with imputation performance. Consider the left-most plot in Figure 7 using OT as a baseline imputer under MAR on our real data sets. Here, we do not have oracle knowledge but assume that the sparseness of the learned DAG implies a coherent DAG. We observe that MIRACLE has the most performance gain when fewer causal parents are identified for the missing variable in the learned DAGs. When MIRACLE is less able to isolate causal parents for prediction, the learned DAGs contains many spurious edges, and MIRACLE only has marginal improvements over the baseline imputer. We note that the gain of MIRACLE is not reproducible via feature selection methods, which are still prone to the spurious correlations in the observed data, as discussed in Section 2.1.

heading: 4.4 MIRACLE Convergence
text: In this subsection, we investigate two dimensions of MIRACLE refinement: (1) baseline imputation quality and (2) sample or instance-wise refinement. Regarding baseline imputation quality, we are interested in understanding the impact of MIRACLE refinement on various baseline imputers that may have disparate performances. In the middle plot of Figure 7, we show MIRACLE applied to various baseline imputers on the Abalone dataset. Similar plots for other datasets can be found in Appendix C. We observe that even though mean imputation starts off with the worst error, after refinement by MIRACLE, we see that all methods converge to similar RMSEs. For the second experiment, we investigate the sample-wise improvement of MIRACLE on the abalone dataset using MissForest as a baseline imputer. On the right-most plot of Figure 7, we observe that a vast majority of the samples are improved by MIRACLE. Note that every point below the diagonal is considered an improvement on an instance over the baseline imputation method. We can see MIRACLE improves the imputation almost universally except for the instances with small errors in the baseline imputation; on these instances, MIRACLE does not inflate their errors by a large margin. Furthermore, we observe that MIRACLE iteratively improves imputation as training progresses (over each epoch) by the observation that the slope of each line decreases with each epoch.

heading: 5 Discussion
text: In conclusion, motivated by the minimax optimization problem (1) arising from interventions on missingness indicators in the m-graph that encode the conditional independencies in the data distribution, we proposed MIRACLE, an iterative framework to refine any baseline missing data imputation to use the causal parents embodied in the estimated m-graphs. MIRACLE learns the causal m-graph as an adjacency matrix embedded in its input layers. We proposed a two-part regularizer based on the causal graph and a moment regularizer based on the missing variable indicators. We demonstrated
that MIRACLE significantly improved the imputations of six baseline imputation methods over a variety of synthetic and real datasets. MIRACLE never hurts performance in the worst-case, and we envision MIRACLE becoming a de facto standard in refining missing data imputation.
There are several limitations we would like to identify as paths for future work. First, any violation of the assumptions in Section 2 may adversely impact the performance of MIRACLE in practice. Second, causal discovery under missing data is an ongoing research area, and therefore MIRACLE may be discovering DAGs with bias introduced from the baseline methods. However, in experiments, MIRACLE still performs well even if it starts with mean imputation. We expect MIRACLE to improve as causal discovery methods under missingness improve. Third, in its current form, MIRACLE is not extensible to scenarios where causality may not be applicable, such as computer vision. Fourth, because of the causal discovery regularizer and network architecture, MIRACLE may have difficulty scaling to very high dimensional data. Lastly, a more general and detailed discussion is needed between our work and the merits of causality and robustness.

heading: Acknowledgements
text: This work was supported by GlaxoSmithKline (GSK), the National Science Foundation (NSF) under grant number 1722516, the Office of Naval Research (ONR), and The Alan Turning Institute (ATI). We thank all reviewers for their invaluable comments and suggestions.

Review summaries:
This paper presents a new framework to impute missing values while preserving the causal structure of the data. The algorithm works by refining the imputations of a baseline method and simultaneously learning the missingness generating mechanisms (which cover different scenarios such as MCAR, MAR, MNAR). The proposed method obtains improved results when compared to the baselines in both synthetic and real experiments.

The authors present MIRACLE, a neural network that learns to impute well thanks a specific regularization scheme. The proposed regularisation is motivated by the idea that when imputation is based on all observed data, spurious correlations can arise while it is not the case when it is based solely on the causal parents. Technically, the weights of the imputation network are regularised so that the dependencies between all imputation functions form a DAG. This is possible thanks to prior work which introduced a score based on adjacency matrices that is equal to zero if and only if the corresponding graph is a DAG.    The neural network proposed by the authors also includes a second regulariser (moment regularizer), this time aimed at matching two different estimators of the first moments of the data. Overall, it minimises a reconstruction loss combined with these two regularisers.    Extensive experiments are conducted on simulated and real data that compare this neural network to various baselines in terms of imputation quality ( as well as performance on downstream tasks and congeniality in Supplementary materials). On simulated data, it is able to improve imputations over all baselines across all missing data mechanisms. On real data, it does not provide improvements in MCAR, but does so for half of the datasets in MAR and MNAR. 

This paper proposes a new method for refining existing baseline missing data imputation algorithms. The method works by starting from an initial estimate from another baseline algorithm, then pass this estimate to two neural networks for refined estimation. The two neural networks are regularized based on intuitions from causal inference, such that the causal missingness of the data can be captured. The refinement process is iteratively run until convergence. The contribution of the paper is a new method for missing data imputation inspired by causal inference concepts.