MIRACLE: Causally-Aware Imputation via Learning Missing Data Mechanisms
Authors: Trent Kyono
Yao Zhang
Alexis Bellot
Mihaela van der Schaar
Venue: NeurIPS 2021 Poster (2021)
Keywords: Missing data
Multiple imputation
Deep learning

Abstract: Missing data is an important problem in machine learning practice. Starting from the premise that imputation methods should preserve the causal structure of the data, we develop a regularization scheme that encourages any baseline imputation method to be causally consistent with the underlying data generating mechanism. Our proposal is a causally-aware imputation algorithm (MIRACLE). MIRACLE iteratively refines the imputation of a baseline by simultaneously modeling the missingness generating mechanism, encouraging imputation to be consistent with the causal structure of the data. We conduct extensive experiments on synthetic and a variety of publicly available datasets to show that MIRACLE is able to consistently improve imputation over a variety of benchmark methods across all three missingness scenarios: at random, completely at random, and not at random.

MIRACLE: Causally-Aware Imputation via Learning Missing Data Mechanisms
Trent Kyono"
Yao Zhang*
University of California, Los Angeles tmkyonoQucla.edt
University of Cambridge yz5550cam. ac
Alexis Bellot
Mihaela van der Schaar
University of Oxford Oxford, United Kingdom
University of Cambridge
University of California, Los Angeles
alexis bellot@eng. ox ac
The Alan Turing Institute mv4720cam ac. uk
Abstract
Missing data an important problem in machine learning practice. Starting from the premise that imputation methods should preserve the causal structure of the data, we develop a regularization scheme that encourages any baseline imputation method to be causally consistent with the underlying data generating mechanism. Our proposal is causally-awe imputation algorithm (MIRACLE). MIRACLE iteratively refines the imputation of baseline by simultaneously modeling the missingness generating mechanism, encouraging imputation to be consistent with the causal structure of the data. We conduct extensive experiments synthetic and a variety of publicly available datasets to show that MIRACLE able to consistently improve imputation over variety of benchmark methods across all three missingness scenarios: at random, completely at random, and not random.
1 Introduction
Missing data is an unavoidable byproduct of collecting data in most practical domains. In medicine, for example, doctors may choose to omit what they deem to be irrelevant information (e.g., some patients may be asked to get comprehensive blood tests while others don '), data may be explicitly omitted by the patient (c.g., avoiding questions smoking status precisely because of their smoking habit) or simply misrecorded in electronic health systems (sce e.g., [3, 12, 28)).
Imputation algorithms can be used to estimate missing values based on data that was recorded, but their correctness depends on the type of missingness. For instance, expanding on the example above, younger patients may also be more likely to omit their smoking status. As illustrated in Figure 1, the challenge is that implicitly conditioning inference on observed data introduces spurious path of correlation between age and the prevalence of smoking that wouldn't 't exist with complete data.
Age
Smoking
Missingness in smoking data
Figure Missingness may introduce spurious dependencies.
Missing data creates shift between the available missing data distribution and the target complete data distribution. is a shift that may be explicitly modeled as missingness indicators in an underlying causal model (i.e., missingness graph as proposed by Mohan et al. [19]) shown in Figure I. The learning problem is one of extrapolation, learning with access to missing data distribution for
Equal contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

True (unknown) adjicency matrix
Baseline
imputation errar
1.45
Estimated adjacency matriti epochs training
Instancee wise improvements
RMSE
After epochs
After 150 epochs
0.15
MIIRACLE reduces error by 62%
0.15
20 50
200
150
200
Training
Figure 2: MIRACLE refines baseline imputation by simultancously learning an m-graph using a bootstrap imputation loop that serves to incrementally regularize predictions with learned causal graph. We plot average lesting error and estimated causal graph as function of training epochs on synthetic data experiment described Section 4. The causal structure (as adjacency matrix) and imputation improvements for each missing value separately (each missing value with a corresponding dot) is shown in the right-most panel.
prediction and inference on the complete data distribution that is, generated from model where all missingness indicators have been intervened on (interventions interpreted in the sense of Pearl [22]) thus graphically removing the dependence between missingness and its causes, and any spurious correlations among its ancestors.
With this causal interpretation, imputation of missing data given variable Y from other observed variables X formulated as problem of robust optimization, -
minimize sup Exx,y>r 0EB PEP
(1)
simultaneously optimizing over the set of distributions P arising from interventions missingness indicators. Causal solutions - i.e. imputation using functions of causal parents of each missing variable in the underlying causal graph are closely-related version of this problem with an uncertainty set Q defined as any distribution arising from interventions on observed variables and variable indicators (see e.g. sections 3.2 and 3.3 in [16]).
sup Exx XX) r) < sup E(xy)~ [(f((X)-) y) PEP
PEQ
since C Q. Our premise is that causal solutions, i.e. minimizing the right- hand-side of (2), are expected to correct for spurious correlations introduced by distribution shift due to missing data and preserve the dependencies of the complete data for downstream analysis.
1.1 Contributions
In this paper, we propose to impute while preserving the causal structure of the data. Missing values in given variable are replaced with their conditional expectation given the realization of its causal parents instead of the more common conditional expectation given all other observed variables, which absorbs spurious correlations.
We propose a novel imputation method called Missing data Imputation Refinement And Causal LEarning (MIRACLE). MIRACLE is general framework for imputation that operates on any baseline (existing) imputation method. visual description is given in Figure 2: given some initial imputation from baseline method, MIRACLE refines its imputations iteratively by learning missingness graph (m-graph) [19] and regularizing the imputation function such that it is consistent with the causal graph generating the data, substantially improving performance. In experiments, we apply MIRACLE to improve six popular imputation methods as baselines. We present detailed simulations to demonstrate synthetic and variety of publicly available datasets from the UCI Machine Learning Repository [8] that MIRACLA ACLE can improve imputation in almost every scenario and never degrades performance across all imputation methods.
1.2
Related work
The literature on imputation large and varied. Still, most imputation algorithms work with the prior assumption that the missing data mechanism is ignorable, in the sense that imputation does

---

not require explicitly modeling the distribution of missing values for imputation [25]. Accordingly, classical imputation methods impute using a joint distribution over the incomplete data that is either explicit or implicitly defined through set of conditional distributions. For example, explicit joint modeling methods include generative methods based on Generative Adversarial Networks [33, 34, 1], matrix completion methods [15], and parametric models for the joint distribution of the data. Missing values are then imputed by drawing from their predictive distribution. The conditional modeling approach [31] consists of specifying one model for each variable and iteratively imputing using estimated conditional distributions. Examples of discriminative methods are random forests [27], autoencoders [11, 10, 14], graph neural networks [35]. distribution matching via optimal transport [20], and multiple imputation using chained equations [6].
In a different line of research, Mohan et al., in a series of papers, see e.g. [19, 18], explicitly considered missing data within the underlying causal mechanisms of the data, Subsequently, a range of related problems has been studied, including identifiability of distributions and causal effects in the presence of missing data, see e.g. [5, 26, 21], testable implications relating to the causal structure using missing data [18], and causal discovery in the presence of missing data [9, 30]. Our focus, in contrast, is algorithmic in nature. We aim to develop an algorithm that improves imputation quality by leveraging causal insights represented as an estimated missingness graph learned from data.
2 Background
The basic semantic framework of our analysis rests on structural causal models (SCMs) (see e.g. Chapter 7 in [22] for more details) explicitly introducing missingness indicators and their functional relationship with other variables, using in part the notation of [19]. We define an SCM M as tuple (X, R. U. F, P) where X is vector of d endogenous variables and U is a vector of exogenous variables." R is the vector of missingness indicators that represent the status of missingness of the endogenous variables X. Precisely, Rg is responsible for the value of a proxy variable Z, of X,, i.e. the observed version of Xj. For example, Zj is equal to X, if the corresponding record is observed (R, 1), otherwise Z is missing (R3 0). F is a set of functions where each fx, fn F decide the values of an endogenous variable X and a missingness indicator variable R, respectively. The function fx takes two separate arguments as parts of X (except X itself) and U, termed as Pax and Ux. That is, X + fx(Pax Ux) and R + (PaR, UR).
The randomness in SCMs comes from the exogenous distribution (u) where the exogenous variables in U are generated independently and are mutually independent. Naturally, through the functions in F, the SCM M induces joint distribution Px (xx over the endogenous variables X, called the endogenous distribution. An intervention on some arbitrary random variables V in X and R. denoted by do) is an operation which sets the value of V to be v, regardless of how they ordinarily determined. For an SCM M, let Mu denote a submodel of M induced by intervention d)(v). The interventional distribution Px (æ: dd(())) induced by do(v) is defined as the distribution over in the submodel My. namely, Px Px.M.. (x) = Px (æ) d)(w)))
Each SCM in the context of missingness is associated with a m-graph G (e.g.. Fig. la), which is directed acyelic graph (DAG) where nodes represent endogenous variables X and missingness indi- cators R, and arrows represent the arguments Pax and Paą of each function fx and fR respectively. By convention, exogenous variables U are often not shown explicitly in the graph.
Assumption (Missingness indicators are not causes) No missingness indicator in R can be the cause of the endogenous variables X, ie., the arguments of the functions generating X.
Assumption (Causal sufficiency) Exogeneous variables U are mutually independent, i.e., all common parents of the endogenous variables are included in X.
Assumption (No self-masking missingness) melk-masking missingness refers to missingness in a variable that is caused by itself. In the m-graph this is depicted by an edge from X, to R2 (as shown in Figure 3 (d)). We assume that there is no such edges in the m-graph.
Assumption (Observed root nodes) The endogenous variables X, such that Pax, 0 (i.e., the root nodes) in the m-graph are always observed (R, with probability 1).
Essentially, X is the ground- trutt features; U is the random noise in the data generating process.
3

---

X1
X3
X1
X
X
X3
X1
X2
X2
R3
X2
R3
R
X2
R2
R1
(a) MCAR graph.
(b) A MAR graph.
(c) A MNAR graph.
(d) Self-masking missingness.
Figure 3: Example graphs. X = (X1, X2, Xa) are endogenous variables and R = (Rt, R2. Ra) are missing data indicators. Red shaded variables are not always observed while white shaded variables are always observed.
We make the four assumptions above throughout the following sections. Assumption and 2 are employed in most related works using m-graphs (see e.g. [18. 19]). Assumption 1 is valid, for example, if R is generated in the data collection process after the variable values are assigned. Consequently, under this assumption, if two endogenous variables of interest X₁ and X2 are not d-separated by some variable X3. they are not d-separated by X3 together with their missingness indicators R1 and R2. We denote an independent relation in a data distribution by "_L" and d- separation in m-graph by "Jld". We assume the data distribution is faithful that the two independencies are equivalent. As shown Figure 3, data is missing completely at random (MCAR) if X XIl. holds in the m-graph, missing at random (MAR) if for any endogenous variable X,. R X-j holds, and missing not at random (MNAR) otherwise, as stated in [19]. If Assumption 3 is violated, Assumption 4 is necessary for imputing all the missing variables from their causal parents. These assumptions are imperative for MIRACLE provide improved imputations by leveraging the causal structure of the underlying data generating process. In our experiments (Section 4), we apply MIRACLE to real-world datasets where these assumptions not guaranteed.
m-graph, meaning
unable to learn the missingness for self mafked variables.
2.1 Why is imputation prone to bias?
The reason for considering the causal structure of the underlying system is that when learning an imputation model from observed data, implicitly conditioning some missingness indicators in R induces spurious dependencies that would not otherwise exist. For example, in a graph X1 + R3 X2. conditioning R3 1 induces dependence between X₁ and X2. In general, the distributions Pxx [ = r) and Px (d)(r) differ unless missingness occurs completely at data
random, and motivates interpretation of the problem as domain generalization, training from one distribution ultimately to be applied on data from a different distribution that, in our case, arises from missing data (i.e., interventions missingness indicators). This shift is not addressed in the imputation methods that only use the feature correlations.
MIRACLE
In this section, propose correct for the shift in distribution due to missing data by searching for causal solutions and explicitly refining imputation methods using a penalty on the induced causal structure. In practice, have i.i.d. realizations of the observed version of X R concatenated as incomplete data matrix X E xx together with missingness indicators concatenated in a matrix We use here the same bold uppercase notation for sets of variables and matrices of observations but their meaning should be clear from the context. Our goal is to impute the unobserved values in X using each variable's causal parents. We define the imputed data X E Raxd
R (0, 1) nxd
X R x R)
where C is the element-wise product of matrices and X is an estimate of the complete data matrix.
3.1 Network architecture
In this section, we describe our approach for estimating X. Let dg S d be the number of partially observed features, i.e., missing for at least one realization. S is the set missing features indices. The imputation network is defined as function f Rd - Rd [0. 1]ds that takes initially imputed dataset X (using an existing baseline imputation method), and returns two quantities:
1. A refined imputation X

---

Basellie Imputation
MIRACLE Refinement
Inpd weight subnneannke
Causal Reg.
Incompiale
Browlin inputed X
Shared hidden leyera
R4
Pias
Flu
F2L
Fhat
L
Loss
Sox
Moment Reg.
Ra
Sochhtasp Impuartio
Figure 4: Network and optimization diagram for MIRACLE.
2. An estimation of the probabilities features Xij being missing, = 1..., n and S.
A depiction of the network architecture and optimization algorithm is shown in Figure 4. The architecture is constructed with respect to the assumptions shown in Section 2. Our model f is decomposed into two sub-networks, f = (f(ip)), f(miis)), responsible for imputing the unobserved data and estimate the probabilities of missingness, respectively. The imputation network has d components, (émp) = (((mp) ......d' one for each variable, and the missingness network has dy components, f(miss) (f) (mmss)) . Jd
p(miss)) Each component, for both networks, has separate
input and output layers but shared hidden layers (of size h). Let (imp) and W (miss) denote the 1j h d weight matrix (we omit biases for clarity) in the input layer of ((mp)) and J (miss)) respectively.
The j-th column of w (((pp) and w(miss) is set to O. Let Wm E R for , M 1, denote the weight matrix of each hidden layer and let W (émp) and W (miis)) be the x h dimensional output layers of each sub-network. The imputation network prediction is given by,
(imp) (x) w Ma p( (W22((WL. (imp) x))),
for 1, ..................... d. And similarly, the missingness network prediction is given by,
Almisa) (x) == o oo (w(mit)) (miss) o(... 6(....(WW) (( (w((miss))))) 1.
M.j
for j..... .d,, where o6-) is the ELU activation function and the sigmoid function. Our network is optimized with respect to three objectives. First, to accurately predict missing values, second, faithfully encode the causal relationships given by the underlying m-graph, and third to satisfy moment constraint of the missing data mechanism on the imputed values.
3.2 Reconstruction loss
The first objective is to train f to correctly reconstruct each feature from the observed data using reconstruction loss,
C
Σ Ox ((mp)) & O rr rr|l
Σ CrossEntropy
p (x(
where x; and (0) are the realized and imputed feature vector of the i-th instance, i; are the ds components of r; that missing for at least one instance. The first loss term is for reconstructing the observed features, and the second loss term is for estimating the probabilities of missingness.
3.3 Causal regularizer
The second objective is to ensure that the dependencies defined by f correspond to DAG over the features X and the missing indicators Rs. which enforces that the learned functional dependencies recover DAG in the equivalence class of causal graphs the observed data. Enforcing the acyclicity of the dependencies induced by continuous function f is originally proposed in [36, 37]. Define binary adjacency matrix E {0, 10 (d+ds)) (d++s)) k.j = 0 (i.e., the 12-norm of the k-th column of the matrix w-(mpp) (miss) is 0) is realistic and sufficient condition for achieving Og. = 0. The adjacency matrix B of the graph induced by the learned f is acyclic if and only if,
R1 } (B) h(B).
33)
5

---

is equal to zero, where h(B) == [B C BJ) (d+ ds) and exp(-) is the matrix exponential.
Remark 1. Existing imputation methods based feature correlations essentially assume undi- rected (non-causal) graph between the features. Further, acyclicity is a realistic and practical assump- tion to make the static datasets collected by human experts. In nature, most data distributions generate their features in some order. In a directed graph, a cycle means a path starts and ends at the same node. This unlikely happen in the data generating process if not considering variables over time, i.e., time-series data. By enforcing acyclicity, MIRACLE only uses the causal parents for imputation, which is less biased by spurious dependencies that only exist in the observed data.
3.4 Moment regularizer
The third objective leverages a set of moment constraints in the missingness pattern to improve imputation. Assume { P(R, derivation holds for MAR or MCAR missingness patterns only. It holds that,
P)R,)) (6, - 6), for some 5 O. The following
R X 5
R Xj E
X3 }=={ E
E
=
Xj, Pan,
[ Xj, Pan,
E (x,),
(4)
where the third equality follows from the MAR assumption (R, 1 X, X Xx))). Under the MCAR assumption, this derivation holds trivially since in that case Ry Xy.
We can use the missingness and imputation networks to enforce the above equality algorithmically. ensuring the left hand side equals the right hand side in the empirical version of (4) follows,
]"
R₂ = Σ j=1
ds
J.SPI - [ymmaan] 1
ds Σ j=l
Σ exgris
n Σ
eigrifat
n 1
(imp)) sbI
(x (2)))
where Eij = -1/11 (x: and S[j] is the j-th element of S, i.e., the index of the j-th missing feature. Minimizing R2 forces the two estimators of E {X,} to match, the stabilized inverse propensity score weighting (SIPW) estimator s3 [24] using the missingness network f(miss)) (in ei²) and the mean estimator 7j,meu using the imputation network f(imp)
We hypothesize this mechanism can improve performance for two reasons. First, the missing data mechanism P(R = PaR,) can be simpler function that takes less samples to learn than the function that generates the feature j. E [Px,) Then the SIPW estimator based on / (mmiss)) will converge to the true mean faster than the estimator based on / Second, in R2,
Remark 2.
55t) the mean estimator using flimp) is based on all the samples; ((6mp) is trained to produce predictions
on the samples with missing feature j for the sake of matching the SIPW estimator. By contrast, without the regularizer R2, ((mp) is solely trained on the samples with observed feature j, and its performance may fail to generalize to data with missing feature j.
3.5 Bootstrap Imputation
Discovering a causal graph requires complete data. However, this is not the case for missing data problems. Because of this, we require that MIRACLE be seeded by another imputation method. Imputed values are iteratively refined by MIRACLE, hence "bootstrapping". to potentially converge to a new imputation that minimizes MIRACLE's objective (including causal and moment regularizers). MIRACLE's objective for optimization is.
C1 + 3, + BgR2.
(5)
where 31 and 33 are hyperparameters that define the strength of regularization. We iteratively update the baseline matrix (0) with new imputed matrix X given by MIRACLE every ten epochs in training. With increasing epochs, stochastic optimization minimizes the loss for the imputed matrices that respect the causal and moment regularization. In theory, this is analogous to supervised training a denoising autoencoder (DAE) [32, 4, 23), but differs only by the fact that "noise" comes from prior or previous imputations. In training DAE, the input samples are corrupted by independent noise

---

Wcon
64N MERGCARO
KWN
07
MICE
Maan MR]
GAN
NICE MIRUMIZI
MEROOTT
MIRUM
MIRGARI
MER/ONNO
MERONIT
SIROCT
1.0
0.2-
0.5
AMSE
Data
1.0
0.2
0.8
RM5E
RM5B
Missingness Ratio
Missingness Ratio
1.0
0.2
0.0
RM5E
RM5E
Feature
Feature
(a) MAR
(b) MNAR
Figure 5: Experiments MAR (left) and MNAR (right) synthetic data in terms of RMSE over varying dataset sizes (top), missingness rates (middle), and feature sizes (bottom). Note show the average error over a variety of DAG instantiations and target variables, thus the magnitude and standard deviation of errors vary significantly between runs. See Appendix B for MCAR results.
with each epoch, yet convergence is still guaranteed [2]. In our experiments, we demonstrate that bootstrap imputation indeed converges on multiple datasets and baseline methods. In Appendix E, we provide experiments on the computational complexity of MIRACLE.
4 Experiments
In this section, validate the performance of MIRACLE using both synthetic and variety of real-world datasets.
1. In the first set of experiments, we quantitatively analyze MIRACLE on synthetic data generated from known causal structure.
2. In the second set of experiments, quantitatively evaluate the imputation performance of MIRACLE using various publicly available UCI datasets [8].
General set-up. We conduct each experiment five times under random instantiations of missingness. We report the RMSE along with standard deviation across each of the five experiments. Unless otherwise specified, missingness applied at rate 30% per feature. For MCAR, this is applied uniformly across all features. For MAR, we randomly select 30% of the features to have missingness caused by another disjoint and randomly chosen of features. Similarly, we randomly select 30% of features to be MNAR. We induce MAR and MNAR missingness using the methods outlined in [33], and we provide more details in Appendix A.2.
We use an 80-20 train-test split. We performed hyperparameter sweep (oseba for 31 and 82 with ranges between le-3 and 100. By default have 31 and B3 set to 0.1 and respectively.
Evaluating imputation. For each subsection below, we present three model evaluations in terms of missingness imputation performance, label prediction performance of prediction algorithm trained on imputed data and the congeniality of imputation models.
Missingness imputation performance is evaluated with the root mean squared error com- paring the imputed missing values with their actual unobserved values.
Label prediction performance of an imputation model is ability improve the post- imputation prediction. By post-imputation, refer to using the imputed data to perform

---

downstream prediction task. To be fair to all benchmark methods, we use the same model (support vector regression) all cases.
The congeniality of an imputation model its ability to impute values that respect the feature-label relationship post imputation. Specifically, we compare, support vector parame- ters, w, learned from the complete dataset with the parameters úŪ learned from the imputed dataset. We report root mean square error ([] w 2/1 for each method. Lower values imply better congeniality [33].
Baseline imputation methods. We apply MIRACLE imputation over a variety of six commonly
used imputation baseline methods: (1) mean imputation using the feature- mean, (2) deep
generative adversarial network for imputation using GAIN [33] (3) k-nearest neighbors (KNN) [29] using the Euclidean distance as a distance metric of each missing sample to observed samples, (4) a tree-based algorithm using MissForest (MF) [27]. (5) a deep neural distribution matching method based on optimal transport (OT) [20], and (6) Multivariate Imputation by Chained Equations (MICE) [6]. For each of the baseline imputation methods with tunable hyperparameters, we used the published values. We implement MIRACLE using the tensorflow library. Complete training details and hyperparameters are provided in Appendix D.
Additional experiments. In Appendix B and C, we also evaluate MIRACLE in terms of providing imputations that improve predictive performance [33, 35J, as well as, in its ability to impute values that respect the feature-label relationship. i.e., congeniality [17, 7]. We also investigate the impact of the graphical missingness location on imputation performance on synthetic data in Appendix G.
4.1 Synthetic data
In this subsection, we evaluate MIRACLE on synthetic data. In doing so, we can control aspects of the underlying data generating process and possess oracle knowledge of the DAG structure.
Data generating process. We generate random Erdos-Renyi graphs with functional relationships from parent to children nodes. At each node, we add Gaussian noise with mean 0 and variance 1. For a complete description of the underlying data generating process, see Appendix A.
Synthetic results. In Figure 5, we show experiments of MIRACLE on synthetic MAR data in terms of RMSE. Our experiments show that MIRACLE is able to significantly improve imputation over each of the baseline imputation methods. Figure shows MIRACLE improves performance over each baseline method across various dataset sizes, missingness ratios, and feature sizes (DAG sizes). We show results for MCAR in Appendix B.
4.2 Experiments real data
In Figure 6 we show experiments of MIRACLE on real data. We perform experiments on several UCI datasets used in [33, 35, 20, 13]: Autism, Life expectancy, Energy, Abalone, Protein Structure, Communities and Crime, Yeast, Mammographic Masses, Wine Quality, and Facebook Metrics. In Figure 6, the improvements of MIRACLE are minimal for MCAR (except for mean imputation). This agrees with our discussion in Section 2. because the baseline imputations are not biased in the MCAR setting where R holds in the m-graph. Conversely for the MAR and MNAR settings, as expected, we observe MIRACLE has an significant improvement on some of the datasets, such Abalone, Autism, Energy and Protein Structure. As discussed in Section MIRACLE can improve the baseline imputation under A Assumptions 1-4, which may not hold in these real-world datasets. Nevertheless, we observe that MIRACLE never degrades performance relative its baseline imputation on any dataset. Furthermore, no baseline imputer is optimal across the datasets. In almost all cases, applying MIRACLE to any baseline results in the lowest error. We show a similar gain for this experiment with MCAR and MNAR in Appendix C. An ablation study on our overall loss is provided in Appendix F.
SSorree code at https:: : ///gtthub. con/vvandeerscaaraaaa//AAAAAC
8

---

MCAR RMSE
0.5
abalone
sutism
energy
protein
li expectancy
community
mammo, masses wine Quality
facebook
2.0
MAR RMSE
1.5
1.0-
0.5
0.0
abalone
autim
protein
life expectancy
facebook
MNAR RMSE
2.0
0.0
auttsm
eneryy
expectancy
facebook
Figure 6: MIRACLE real data. MIRACLE improves all baselines across MCAR (top), MAR (middle), and MNAR (bottom). Results for predictive error and congeniality can be found in the Appendix C.
4.3 Causal discovery and imputation performance
In our experiments, we observe a positive correlation between the quality of learned DAGs (and causal parents) with imputation performance. Consider the left-most plot in Figure 7 using OT as baseline imputer under MAR on our real data sets. Here, we do not have oracle knowledge but assume that the sparseness of the learned DAG implies a coherent DAG. We observe that MIRACLE E has the most performance gain when fewer causal parents are identified for the missing variable the learned DAGs. When MIRACLE is less able to isolate causal parents for prediction, the learned DAGs contains many spurious edges, and MIRACLE only has marginal improvements over the baseline imputer. We note that the gain of MIRACLE is not reproducible via feature selection methods, which are still prone to the spurious correlations in the observed data, as discussed in Section 2.1.
4.4 MIRACLE Convergence
In this subsection, we investigate two dimensions of MIRACLE refinement: (1) baseline imputation quality and (2) sample or instance- wise refinement. Regarding baseline imputation quality, we interested in understanding the impact of MIRACLE refinement on various baseline imputers that may have disparate performances. In the middle plot of Figure 7, we show MIRACLE applied to various bascline imputers on the Abalone dataset. Similar plots for other datasets can be found in Appendix C. We observe that even though mean imputation starts off with the worst error, after refinement by MIRACLE, we see that all methods converge to similar RMSES. For the second experiment, we investigate the sample-wisg wise improvement of MIRACLE on the abalone dataset using MissForest as baseline imputer. On the right-most plot of Figure 7, we observe that vast majority of the samples are improved by MIRACLE. Note that every point below the diagonal is considered an improvement on an instance over the baseline imputation method. We can see MIRACLE improves the imputation almost universally except for the instances with small errors the baseline imputation: on these instances, MIRACLE does not inflate their errors by a large margin. Furthermore, we observe that MIRACLE iteratively improves imputation as training progresses (over each epoch) by the observation that the slope of each line decreases with each epoch.
Discussion
In conclusion, motivated by the minimax optimization problem (1) arising from interventions on missingness indicators in the m-graph that encode the conditional independencies in the data distribu- tion, we proposed MIRACLE, an iterative framework to refine any baseline missing data imputation to use the causal parents embodied in the estimated m-graphs. MIRACLE learns the causal m-graph as an adjacency matrix embedded in input layers. We proposed two-part regularizer based on the causal graph and a moment regularizer based on the missing variable indicators. We demonstrated

---

1.8
Baseline
1.0
Epoch
improvement MIRAGLE
80-
expeciancc
Moalone
MIRACLE RMSE
1.5
KNN
Mean
GAIN
MIRACLE Instance wise RMSE
0.8
20
50
150
60-
communtty
1.2
OT
1.0
NICE
40-
Phoeoook
0.8-
0.4
20-
0.2-
0.4
0.2
0.0
0.2
0.5
0.6
100
250
0.0
0.2
3.4 0.6 0.5
1.0
parents
Epoch
Baseline instance-wise RMSE
Figure 7: (Left) Analysis of MIRACLE w.r.f causal parents real data. MIRACLE has the most gain when we have identified sparse set of causal parents. When many features are identified as causal parents, imputation performance degrades. (Mid) Convergence MIRACLE across various baseline imputers. the abalone dataset, we show that MIRACLE converges to consistent RMSE regardless of baseline imputation. (Right) Sample- wise RMSE for MIRACLE across various epochs. MIRACLE is applied to refine MissForest imputations, demonstrating that error reduced in sample wise basis. Note: anything below the diagonal, is an improvement over the baseline imputations.
that MIRACLE significantly improved the imputations of six baseline imputation methods over variety of synthetic and real datasets. MIRACLE never hurts performance in the worst-case, and we envision MIRACLE becoming de facto standard in refining missing data imputation.
There are several limitations we would like identify paths for future work. First, any violation of the assumptions in Section 2 may adversely impact the performance of MIRACLE in practice. Second, causal discovery under missing data is an ongoing research area, and therefore MIRACLE may be discovering DAGs with bias introduced from the baseline methods. However, in experiments, MIRACLE still performs well even if starts with mean imputation. We expect MIRACLE to improve as causal discovery methods under missingness improve. Third, in its current form, MIRACLE is extensible scenarios where causality may not be applicable, such computer vision. Fourth, because of the causal discovery regularizer and network architecture, MIRACLE may have difficulty scaling to very high dimensional data. Lastly, more general and detailed discussion is needed between our work and the merits of causality and robustness.
Acknowledgements
This work was supported by GlaxoSmithllnee (GSK), the National Science Foundation (NSF) under grant number 1722516, the Office of Naval Research (ONR), and The Alan Turning Institute (ATT). We thank all reviewers for their invaluable comments and suggestions.
References
[1] A. Allen and W. Li. Generative adversarial denoising autoencoder for face completion, https: Avery Wenchen/ 2016.
/ //www.cc. gatech. w
[2] Sanjeev Arora, Aditya Bhaskara, Rong Ge, and Tengyu Ma. Provable bounds for learning some deep representations. In International conference machine learning, pages 584 592. PMLR, 2014.
[3] Melanie Bell, Mallorie Fiero, Nicholas Horton, and Chiu-Hsich Hsu. Handling missing data in rets; review of the top medical journals. BMC medical research methodology, 14(1):1-8, 2014.
[4] Yoshua Bengio, Li Yao, Guillaume Alain, and Pascal Vincent. Generalized denoising auto- encoders generative models. In Proceedings of the 26th International Conference on Neural Information Processing Systems Volume 1, NIPS' 13, page 899-907, Red Hook, NY, USA, 2013. Curran Associates Inc.
[5] Rohit Bhattacharya, Razieh Nabi, Ilya Shpitser, and James Robins. Identification in missing data models represented by directed acyelic graphs. Proc. of Conference on Uncertainty in Artificial Intelligence (UAI), 06 2019.
10

---

[6] Stef Buuren and Catharina Proothuis-Oudshomm Mice: Multivariate imputation by chained equations in r. Journal of Statistical Software, 45, 12 2011.
[7] Yi Deng, Changgee Chang, Moges Ido, and Qi Long. Multiple imputation for general missing data patterns in the presence of high-dimensional data. Scientific Reports, 6:21689, 02 2016.
[8] Dheeru Dua and Casey Graff. UCI machine learning repository, 2020.
[9] Alexander Gain and Iya Shpitser. Structure learning under missing data. Proceedings of machine learning research, 72:121, 2018.
[10] Lovedeep Gondara and Ke Wang. Multiple imputation using deep denoising autoencoders. 05
2017.
[11] Yu Gong, Hossein Hajimirsadeghi,, Jiawei He, Megha Nawhal, Thibaut Durand, and Greg Mori. Variational selective autoencoder. In Cheng Zhang, Francisco Ruiz, Thang Bui, Adji Bousso Dieng, and Dawen Liang, editors, Proceedings of The 2nd Symposium on Advances Apprexi- mate Bayesian Inference, volume 1 118 of Proceedings of Machine Learning Research, pages 1-17. PMLR, 08 Dec 2020.
[12] Janus Christian Jakobsen, Christian Gluud, Jømn Wetterslev, and Per Winkel. When and how should multiple imputation be used for handling missing data in randomised clinical trials-a practical guide with flowcharts. BMC medical research methodology. 17(1):1-10, 2017.
[13] Trent Kyono, Yao Zhang, and Mihaela van der Schaar. Castle: Regularization via auxiliary causal graph discovery. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1501-1512. Curran Associates, Inc., 2020.
[14] Pierre-Alexandre Mattei and Jes Frellsen. MIWAE: Deep generative modelling and imputation of incomplete data sets. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 4413-4423, PMLR, 09-15 Jun 2019.
[15] Rahul Mazumder, Trevor Hastie, and Robert Tibshirani. Spectral regularization algorithms for learning large incomplete matrices. Journal of Machine Learning Research, 1180))2227-2332. 2010.
[16] Nicolai Meinshausen. Causality from distributional robustness point of view. In 2018 IEEE Data Science Workshop (DSW), pages 6-10. IEEE, 2018.
[17] Xiao-Li Meng. Multiple-imputation inferences with uncongenial sources of input. Statistical Science, 9()):338-555 -5833, 1994.
[18] Karthika Mohan and Judea Pearl. On the testability of models with missing data. In Artificial Intelligence and Statisties, pages 643-650,220 2014.
[19] Karthika Mohan, Judea Pearl, and Jin Tian. Graphical models for inference with missing data. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26, pages 1277-1285. Curran Associates, Inc., 2013.
[20] Boris Muzellec, Julie Josse, Claire Boyer, and Marco Cuturi. Missing data imputation using optimal transport. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th Inter- national Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 7130-7140, Virtual, 13-18 Jul 2020. PMLR.
[21] Razieh Nabi, Rohit Bhattacharya, and 1lya Shpitser. Full law identification graphical models of missing data: Completeness results. In Proc. of International Conference on Machine Learning (ICML), 2020.
[22] J. Pearl. Causality, Causality: Models, Reasoning, and Inference. Cambridge Univ. Press, 2009.

---

[23] Arn Pretorius. Steve Kroon, and Herman Kamper. Learning dynamics of linear denoising autoencoders. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4141-4150 PMIR, 10-15 Jul 2018.
[24] James Robins, Mariela Sued, Quanhong Lei-Gomez, and Andrea Rotnitzky. Comment: Per- formance of double-robust estimators when" inverse probability" weights are highly variable. Statistical Science, 22(():544-55 2007.
[25] Shaun Seaman, John Galati, Dan Jackson, and John Carlin. What is meant by" missing at random"? Statistical Science, pages 257-268, 2013.
[26] Iya Shpitser, Karthika Mohan, and Judea Pearl. Missing data as causal and probabilistic problem. In Proc. of Conference on Uncertainty in Artificial Intelligence (UAI), 2015.
[27] Daniel J. Stekhoven and Peter Bühlmann, Missforest non-parametric missing value imputation for mixed-type data. Bioinformatics, 28 1:112-8, 2012.
[28] Jonathan AC Sterne, Ian R White, John B Carlin, Michael Spratt, Patrick Royston, Michael G Kenward, Angela M Wood, and James R Carpenter. Multiple imputation for missing data in cpidemiological and clinical research: potential and pitfalls. Bmj, 338, 2009.
[29] Olga Troyanskaya, Mike Cantor, Gavin Sherlock, Trevor Hastie, Rob Tibshirani, David Botstein, and Russ Altman. Missing value estimation methods for dna microarrays. Bioinformatics, 17:520-525, 07 2001.
[30] Ruibo Tu, Cheng Zhang, Paul Ackermann, Karthika Mohan, Hedvig Kjellström, and Kun Zhang. Causal discovery in the presence of missing data. In The 22nd International Conference on Artificial Intelligence and Slutistics, pages 1762-1770. PMLR, 2019.
[31] S. van Buuren. Flexible Imputation of Missing Data. Chapman & Hall/CRC Interdisciplinary Statistics. CRC Press LLC, 2018.
[32] Pascal Vincent, H. Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in deep network with local denoising criterion. J. Mach. Learn. Res., 11:3371-3444 2010.
[33] Jinsung Yoon, James Jordon, and Mihacla van der Schaar. Gain: Missing data imputation using generative adversarial nets. In ICML, 2018.
[34] Seongwook Yoon and Sanghoon Sull. Gamin: Generative adversarial multiple imputation network for highly missing data. In The IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[35] Jiaxuan You, Xiaobai Ma, Daisy Yi Ding, Mykel Kochenderfer, and Jure Leskovec. Handling missing data with graph representation learning. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2020.
[36] Xun Zheng, Bryon Aragam, Pradeep Ravikumar, and Eric P. Xing. DAGS with NO TEARS: Continuous Optimization for Structure Learning. In Advances in Neural Information Processing Systems, 2018
[37] Xun Zheng, Chen Dan, Bryon Aragam, Pradeep Ravikumar, and Eric Xing. Learning sparse nonparametric dags. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Muchine Learning Reseurch, pages 3414-3425. PMLR, 26-28 Aug 2020.
12


Reviews:
[{"review_id": "uOsNvWzAJq", "reviewer": "Reviewer_4iEJ", "summary": "The authors present MIRACLE, a neural network that learns to impute well thanks a specific regularization scheme. The proposed regularisation is motivated by the idea that when imputation is based on all observed data, spurious correlations can arise while it is not the case when it is based solely on the causal parents. Technically, the weights of the imputation network are regularised so that the dependencies between all imputation functions form a DAG. This is possible thanks to prior work which introduced a score based on adjacency matrices that is equal to zero if and only if the corresponding graph is a DAG.  \n\nThe neural network proposed by the authors also includes a second regulariser (moment regularizer), this time aimed at matching two different estimators of the first moments of the data. Overall, it minimises a reconstruction loss combined with these two regularisers.\n\n  Extensive experiments are conducted on simulated and real data that compare this neural network to various baselines in terms of imputation quality ( as well as performance on downstream tasks and congeniality in Supplementary materials). On simulated data, it is able to improve imputations over all baselines across all missing data mechanisms. On real data, it does not provide improvements in MCAR, but does so for half of the datasets in MAR and MNAR.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**:\n- The idea of using a regularizer that promotes the dependencies among imputation functions to follow a DAG is original.\n- Experiments are extensive. In the simulations, the number of samples, features, and missing data mechanisms (MCAR, MAR, MNAR) are varied. Experiments are also carried on 10 real datasets, with varied missing data mechanisms.\n- The baselines include recent work (Gain and Optimal-transport based imputations) on top of classical baselines.\n- The improvements in terms of imputation performance are clear in simulations and part of the real data.\n\n\n**Weaknesses and questions**:\n- Paragraph 3.3 (Causal Regularizer) is at the heart of the paper. Yet, it is very short and a few things are not clear to me. \nThis paragraph describes how the weights of the imputation network can be regularised so that the dependencies defined by f correspond to a DAG).\n\t- From my understanding, $\\partial_k f_j$ is a number and not a vector. Is it the case? If yes the norm used around the partial derivative is a bit misleading.  Also the phrase ‘which can be obtained from the l2-norm of the k-th row…’ is unclear to me. It is a sufficient but non necessary condition for the derivative to be zero? And wouldn’t it be the column instead of the row?\n    - Finally, the regularizer 1 (for ‘DAGness’) is about the adjacency matrix B but it is unclear to me what quantity is used for B_{k, j} as a function of the weights of the imputation network. I understand it is a quantity that depends on the weights of the first layer of $f_j$, but I do not understand exactly what quantity and why.\nIt would be helpful to explain this part a but more.\n\n- It is unclear why MIRACLE works on MCAR simulated data while, as stated by the authors on the real data part, it is not expected to improve performance on MCAR data. On real data, MIRACLE does not improve results compared to baselines in MCAR. The authors explain that this finding agrees the discussion in section 2.1, which states that in MCAR the baselines are not biased. However, the simulations in MCAR in appendix show that MIRACLE brings improvement even if it is MCAR. It seems inconsistent at first sight with the explanation given for the real data. What could be the explanation for this?\n\n- Some important details about the experiments are not given.\n    - A downside of using two regularisers is that two hyperparameters need to be set. How are they set in the experiments?\n    - What sample sizes are used for the train and test sets?\n    - In the MAR and MNAR simulations, I understand that 30% of features have 30% missingness, is it correct? That would lead to a much smaller amount of missing values than in MCAR, in which I understand all features have 30% missing values.\n\n-  I think it is important to understand the relative importance of the two regularizers, which are very different in nature. It is nice to have the ablation study for the real data, however I think it would give more insights into the relative importance of the regularisers on the simulated data. Notably, should we expect that the moment regularizer, which is derived for the MAR case and not appropriate for the MNAR case, help in the MNAR case?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "UMedYsjousi", "reviewer": "Reviewer_XE6q", "summary": "This paper presents a causally-aware imputation algorithm (MIRACLE) which refines imputation of a baseline by simultaneously modeling the missingness generating mechanism. Extensive experiments on synthetic and a variety of publicly available datasets with several baseline comparisons are presented.", "questions": "", "limitations": "", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Very well done paper that leverages the modeling of the missing data mechanism. Along with knowledge of a causal graph, this kind of understanding of the data generating process is strong and should be condoned in data analytic systems.\n*The paper is very thorough, figures are clear and elucidate well \n*Very through experiments are presented, systematically varying feature and data size\n*It would be helpful to see more discussion on the premise, that causal solutions are expected to correct for spurious correlations introduced by distribution shift due to missing data and preserve the dependencies of the complete data for downstream analysis.  If spurious variables are included (e.g. as parent variables), which can happen in data analyses, then how would those be de-coupled?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations not included.\nAuthors are suggested to identify limitations of causal graphs and the types of data that can be viewed in such modeling approaches.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "SUVIjn79eoR", "reviewer": "Reviewer_qmy6", "summary": "This paper proposes a new method for refining existing baseline missing data imputation algorithms. The method works by starting from an initial estimate from another baseline algorithm, then pass this estimate to two neural networks for refined estimation. The two neural networks are regularized based on intuitions from causal inference, such that the causal missingness of the data can be captured. The refinement process is iteratively run until convergence. The contribution of the paper is a new method for missing data imputation inspired by causal inference concepts.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The proposed method is inspired by causal inference, and this is less explored in the literature. Successful application of causal inference to missing data would be a significant step.\n\n+ The paper ran extensive experiments and overall the benefits of the algorithm is demonstrated (see some minor points below)\n\n+ The paper is well organized and easy to follow.\n\n- I am a little bit concerned about the identifiability problem. As the paper mentioned in Line 205 itself, \"Discovering a causal graph requires complete data. However, this is not the case for missing data.\" From my understanding, identifying causal structures requires access to interventions data, and I am not sure how the proposed approach connects to the identifiability problem, and also, how the four assumptions help define the identifiability. For example, is any theorems from [18] that could help explain why the proposed method can help recover the missing data/ missing mechanism? I think the paper lacks detailed discussion on this part.\n\n- Following the previous point, if we are trying to recover the causal structures, is it possible to have some simulation studies to understand how well we are approximating the underlying causal graph? I think Figure 2 is trying to show the adjacency graph is improved over the course of the training. When we are synthesizing the missingness of the data, it appears possible to report how well we are learning the underlying graph by using the matrix B and compare to our synthesis.\n\n- (Minor) It would be helpful for readers if the three missing data mechanisms are explicitly discussed.\n\n- (Minor) The paper discussed about robust optimization in the introduction, but I think this discussion is forgotten in the later sections. How does the proposed approach connects to robust optimization is an interesting topic.\n\n- For the experiments part, some of the results have really large error bars, for example, Figure 10. b), Figure 12 [life expectancy, community, yeast] in Appendix, and looks like the results would not be significant if you compare the baseline vs baseline + MIRACLE. I am wondering whether this provides any insight on the proposed algorithm\n\n[18] Karthika Mohan, Judea Pearl, and Jin Tian. Graphical models for inference with missing data. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26, pages 1277–1285. Curran Associates, Inc., 2013.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- On the computational complexity, I think the paper doesn't address it well. There are some discussions in Appendix E., \"Computational time scales linearly with increasing the number of input samples.\". However, a more important aspect, from my point of view, is the complexity w.r.t to the feature size. Since a matrix exponential is used in 3.3 Causal regularizer, the complexity is $O(d^3)$, and it would probably be very challenging to apply in a large scale applications.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "2A5RozLFZN_", "reviewer": "Reviewer_c6qR", "summary": "This paper presents a new framework to impute missing values while preserving the causal structure of the data. The algorithm works by refining the imputations of a baseline method and simultaneously learning the missingness generating mechanisms (which cover different scenarios such as MCAR, MAR, MNAR). The proposed method obtains improved results when compared to the baselines in both synthetic and real experiments.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, I think this is a good paper. It addresses a very timely problem of great interest to the machine learning community, i.e. the interplay between missing data imputation and the causal structure of the dataset (including the missingness indicators). The paper is well written and motivated. The proposed methodology is clear (in particular Figure 4 is useful to understand the network and optimization process), it seems sound, and it is different to previous approaches. Also, the experimental validation is well structured and is convincing. \n\nPerhaps what I most miss is a deeper evaluation of the estimated m-graph. The focus of the experimental section is to show that MIRACLE reduces the test RMSE compared to the baseline. Only Figure 7 (left) presents a small discussion on the causal relationships found, although it is still linked to the improvement in terms of RMSE. My suggestion would be to analyse the m-graph estimated in the synthetic experiment, and compare it with the available ground truth. Many interesting questions can be answered there (e.g. are the causal relationships well reconstructed? is the causal regularizer really having an effect on the graph?).\n\nSome other questions and comments that seem relevant:\n\n* MIRACLE is a framework to refine an _existing_ imputation method by learning the causal structure of the dataset. I wonder whether it would be possible to develop (or whether it exists) some algorithm that provides the same functionality but _directly_, i.e. not starting by an existing imputation method that may introduce some bias. That is, I am thinking of an algorithm that simultaneously learns two things: 1) the causal structure of a dataset and 2) how to impute missing values. \n\n* I was wondering why $[B]_{k,j}$ is defined like that in lines 173-174. I understand that, by doing the partial derivative, you are checking whether the j-th component depends on the k-th variable. And this is indeed what you want for the corresponding item of the adjacency matrix. If that is the case, it may be worth stating it explicitly.\n\n* In the experimental section, there are some incorrect references to the Appendix. For instance, in the last sentence of Figure 5, it says \"See Appendix B for MCAR and MNAR results\". However, MNAR results are indeed in Figure 5. This wrong references happen several times throughout the experimental section, so please check it.  \n\nUPDATE\n\nAfter reading the rest of reviews and the authors response, I think that this paper is already in good shape to be accepted. Therefore I keep my score (7). ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers have provided thoughtful and constructive comments. They have responded to the authors' feedback and the most active reviewers who are championing this manuscript continue to lean towards acceptance. I hope the authors will take the reviewers' comments to heart and encourage them to incorporate their thoughts in preparing the camera-ready version of their manuscript.
Decision: Accept (Poster)