Improving the imputation of missing data with Markov Blanket discovery
Authors: Yang Liu
Anthony Constantinou
Venue: ICLR 2023 poster (2023)
Keywords: Feature selection
Imputation
Markov Blanket discovery

Abstract: The process of imputation of missing data typically relies on generative and regression models. These approaches often operate on the unrealistic assumption that all of the data features are directly related with one another, and use all of the available features to impute missing values. In this paper, we propose a novel Markov Blanket discovery approach to determine the optimal feature set for a given variable by considering both observed variables and missingness of partially observed variables to account for systematic missingness. We then incorporate this method to the learning process of the state-of-the-art MissForest imputation algorithm, such that it informs MissForest which features to consider to impute missing values, depending on the variable the missing value belongs to. Experiments across different case studies and multiple imputation algorithms show that the proposed solution improves imputation accuracy, both under random and systematic missingness.

Published as a conference paper at ICLR 2023
IMPROVING THE IMPUTATION OF MISSING DATA WITH MARKOV BLANKET DISCOVERY
Yang Liu, Anthony C. Constantinou Machine Intelligence and Decision Systems (MInDS) Research Group Queen Mary University of London {yangliu, a. www.onntttttt.m nou ) qmul aac. uk
ABSTRACT
The process of imputation of missing data typically relies on generative and re- gression models. These approaches often operate on the unrealistic assumption that all of the data features are directly related with one another, and use all of the available features to impute missing values. this paper, we propose a novel Markov Blanket discovery approach to determine the optimal feature set for a given variable by considering both observed variables and missingness of partially observed variables to account for systematic missingness. We then incorporate this method to the learning process of the state-of-thh-aa MissForest imputation algorithm, such that it informs MissForest which features to consider impute missing values, depending on the variable the missing value belongs to. Experi- ments across different case studies and multiple imputation algorithms show that the proposed solution improves imputation accuracy, both under random and sys- tematic missingness.
INTRODUCTION
Dealing with missing data values represents common practice across different scientific domains, especially in clinical (Little et al. 2012 Austin et al., (021). genomics (Petrazzini et al. 2021) and ecological studies (Alsaber et al. 2021 Zhang Thorburn 2022). It represents a problem that can be difficult to address accurately, and this is because missingness can be caused by various known and unknown factors, including machine fault, privacy restriction, data corruption, inconsistencies in the way data are recorded, as well as purely due to human error.
Rubin (976) categorised the problem of missing data into three classes known as Missing Com- pletely At Random (MCAR), Missing At Random (MAR) and Missing Not At Random (MNAR). We say data MCAR when the missingness purely random, i.e., the missing mechanism is in- dependent of both the observed and unobserved values. On the other hand, data is MAR when the missingness is dependent on the observed values but independent of the unobserved values given the observed values; implying that MAR data be effectively imputed by relying on observed data alone. Lastly, data is said to be MNAR when it is neither MCAR nor MAR and hence, missingness dependent on both the observed and unobserved values. While it is tempting to simply remove data rows that contain empty data cells, a process often referred to as list-wise deletion or complete case analysis, past studies have shown that such an approach is ineffective since it tends lead to poorly trained models (Wilkinson) 1999. Baraldi & Enders] 2010). On this basis, the problem of missingness is typically handled by imputation approaches which estimate the missing values, often using regression generative models, and return complete data set.
The imputation algorithms are often classified as either statistical or machine learning methods (Lin & Tsai] 2020). Statistical imputation methods include Mean/Mode, which is one of the simplest methods where the imputation is derived by the mean or mode the observed values found in the same data column. A more advanced statistical method is the algorithm (Honaker et al., 201T). EM computes the expectation sufficient statistics given the observed data at the E-step (Expectation), and then maximizes likelihood the M-step (Maxi- mization). It iterates over these two steps until convergence, at which point the converged pa-
(EM)
rameters used along with the observed data impute missing values. Another statistical al-

---

Published as a conference paper at ICLR 2023
gorithm is the one proposed by Hastie et al. (2015), called softlmpute, which treats imputation as a matrix completion problem and solves it by finding a rank-restricted singular value decompo- sition. Multiple imputation is another popular statistical method for handling missing data, and considers the uncertainty of missing values. Some classic multiple-impuaation algorithms include the Multivariate Normal Imputation (MVNI) (Lee & Carlin 2010), Multiple Imputation by Chained Equations (MICE) (Van Buuren & Groothuis-Oudshone 2011). and Extreme Learning Machine (ELM) [Sovilj et al. 2016).
On the other hand, one of the earliest imputation methods that come from the Machine Learning (ML) field include the k-nearest neighbour (k-NN) (Zhang 2012), which imputes empty cells ac- cording to their k-nearest observed data points. A well-established ML imputation algorithm is MissForest (Stekhoven
Bühlmann 2012), which trains a Random Forest (RF) regression model recursively given the observed data, for every variable containing missing values, and uses the trained RF model to impute missing values. Recently, deep generative networks have also been used for imputing missing data values. Yoon et al. (0018) proposed the Generative Adversarial Imputation Nets (GAIN) algorithm which trains the generator to impute missing data and the dis- criminator to distinguish original data and imputed data, and shown to have higher imputation accuracy compared to previous approaches. Other ML techniques used for imputation include the optimal transport (Muzellec et al. 2020), neural network with causal regularizer (Kyono et al. 202T], and automatic model selection (Jarrett et al. 2022).
All of the aforementioned algorithms assume that all the variables in the data correlate with each other, and use all the variables to impute the missing values. Considering all of the data variables increases the risk of over-fitting, but which can be minimised through LI and L2 regularization methods often employed by ML algorithms. However, regularization leads to models that tend to lack interpretability and theoretical guarantees of correctness. Because this paper focuses on inter- pretable models, such as those produced by structure learning algorithms,, we shall focus on causal feature selection which maintains interpretability. rather than regularization. This is also partly moti- vated by Dzulka Ininniee & Sallehuddin (2019) who showed that using uncorrelated variables to impute missing values not only decreases learning efficiency. but also degrades imputation accuracy. On this basis, it has recently been suggested to include feature selection phase that prunes off poten- tially unrelated variables, for each variable containing missing values, prior to imputation (Bu et al.) 2016 Liu et al., 2020; Hieu Nguyen et al. 2021).
Relevant studies that focus on feature selection for imputation include the work by Doquire & Ver- leysen (2012) who used Mutual Information (MI) measure the dependency between variables. They used a greedy forward search procedure to construct the feature subset, which is an itera- tive process that constructs feature sets that maximise MI with the dependent variable. [Sefidian & Daneshpour (2019) also estimate the dependency between variables using MI, and chose to select set of variables that increase MI above given threshold, as the features of a given dependent variable. On the other hand, the algorithm proposed by Dzulkalnine & Sallehuddin (2019) applies a fuzzy Principle Component Analysis (PCA) approach the complete data cases to remove irrel- evant variables from the feature set, followed by a SVM classification feature selection task that returns the set of features that maximise accuracy on the dependent variable. Lastly, evolutionary optimisation algorithms have also been adopted for feature selection in imputation, and include dif- ferential evolution (Tran et al. 2018), genetic algorithms (Awawdeh et al. 2022). and particle swarm optimisation (Jin et al. 2022).
Recently, causal information has also been adopted to feature selection for missing data imputation. Kyono et al. (2021) proposed to impute missing values of variable given its causal parents derived from the weights of the input layer in the neural network. Similarly, Yu et al. (2022) proposed the MimMB framework that learns Markov Blankets (MBs) be used for feature selection in imputa- tion, which is an iterative process that learns MBs from the imputed data and updates the learned MB after each iteration. Note that while MimMB is related our work, since we also use MB con- struction for feature selection, an important distinction between the two is that MimMB combines MBs with imputed data whereas, as we later describe in Section 3 the learning phase of MBs that we propose is separated from imputation, accounts for partially observed variables, and improves computational efficiency.
In this paper, we use the graphical expression of missingness proposed by Mohan et al. (2013), known as m-graph, which is graph that captures observed variables in conjunction with the possible
2

---

Published as a conference paper at ICLR 2023
causes of missingness as parents of the partially observed variables. We first show that an original version of the Grow and Shrink (GS) algorithm by MMrgaritis (2003) is capable of discovering the MBs in m-graphs containing partially observed variables, when applied to test-wise deleted data. Because this approach relies on CI tests with large conditioning sets, we modify GS such that the number of conditioning sets considered for CI tests is reduced. We provide proof that the modified GS is capable of discovering the MBs of partially observed variables in m-graphs, under the same assumptions as with the original GS. We then propose a new imputation algorithm, which we call Markov-BB Markov-Blanket Miss-Forest (MBMF), that combines the modified GS with the state-of- are-art MissForest (MF) imputation algorithm. We evaluate the effectiveness of MBMF on both synthetic and real-world data sets. The empirical experiments show that MBMF outperforms MF, and other relevant state-of-the-an imputation algorithms,, under most experiments.
PRELIMINARIES
2.1 BAYESIAN NETWORK
A Bayesian network (G.P) is probabilistic graphical model represented by Directed Acyelic Graph (DAG) G = (V, E) and associated probability distributions P over V. In DAG g, a path is sequence of distinct nodes such that every pair of nodes is adjacent in G. A node V; is called collider on a path p if at least two of its neighbouring nodes on are parents of V; in g. We denote node Vj as a descendant of V, if there is path from VÚ to V3 such that all arrowheads of the edges on that path are from VÚ to Vj. A key concept of DAG is d-separation, which defines the conditional independence (CI) between variables in DAG.
Two variables X and Y are d-separated conditional on a variable set Z if every path between X and Y has node W that satisfies one of the following two conditions: i) W is not a collider and W E Z, or ii) W is collider and none of W or its descendant are in Z (Pearl 1988).
D-separation:
Conditional independence entailed by given DAG via d-separation is not always equivalent to the conditional independence of the corresponding probability distribution. However, we assume the Markov and faithfulness conditions described below, where the DAG and the corresponding distribution express the same set of conditional independence.
Markov condition: Given a DAG G over V, every variable in V independent of its non- descendants conditional on its parents.
Given DAG 9 over V. probability distribution is faithful to G if and only if the conditional independence relationships in P are exactly the same as the independences entailed by d-separation G.
Faithfulness condition:
Given the faithfulness condition, a variable is conditionally independent of all the other variables given its MB, which contains all its parents, children and parents of its children. We denote MB of variable V; as MB (V:).
2.2 MISSINGNESS MECHANISM AS AN M-GRAPH
We adopt the graphical representation of the mechanism of missing data known as m-graph, pro- posed by Mohan al. (2013), and which makes a slightly stronger assumption on MAR and MNAR scenarios compared to the definition proposed by Rubin (1976). Given a missing data set, the ob- served variables V can be partitioned into fully observed variables V and partially observed vari- ables V In an m-graph, there is an auxiliary indicator variable R; for each partially observed variable VG E V that specifies the missingness of V,, such that R, 1 when Vú is missing and
R. O when V; is observed. We have MCAR if R IL Vo U VmM MAR if R IL Vm Vo, otherwise MNAR. We denote R,s over a set of variables S as R& = Uves {R, V E Vm). We also make the Assumption and Assumption 2 for indicator variables, as described below and based on (Mohan et al. 2013).
Assumption 1. No missing indicator variable R, can be a parent of observed variables or other indicator variables, i.e., R; could only be leaf node in m m-graph.
Assumption 2. In an m- graph no edge can exist from a partially observed variable V; to its corre- sponding indicator variable R,.
3

---

Published as a conference paper at ICLR 2023
R
Ri
Rı
O
V2
V3
V22
V3
V2
V
VE
VI
VS
VE
VI
V
VE
VI
VE
R3
Vi
R3
V
R
O
(a) MCAR
(b) MAR
(c) MNAR
Figure 1: M-graphs under MCAR, MAR and MNAR conditions respectively. Shaded nodes repre- sent partially observed variables.
Given Assumption if two variables V; and Vj are d-separated by a variable set S, they are still d-separated given SUR. Given Assumption we exclude the situation that there is causal relation between Vi and R2 in order to avoid performing CI test between V/2 and Rg. Figure presents three m-graph examples under different mechanisms of missingness.
MARKOV BLANKET BASED FEATURE SELECTION FOR IMPUTATION
Given the description of the m-graph and causal faithfulness assumption, the problem of feature selection under incomplete data can be converted into MB discovery problem over m-graphs that contain partially observed variables and missing indicators") Because of the possible causal links between partially observed variables V and indicator variables R (i.e., in the case of MNAR), the MB of partially observed variable is likely to contain both observed and indicator variables. For example, in Figure II the MB of V1 is (V2, V3, V1, V3, Rs).
We show that the Grow and Shrink (GS) algorithm with test-wise deletion is capable of discovering the m-graph MB of any variable from incomplete data. Here, unlike list-wise deletion which re- moves all data rows containing at least one missing value, we use test-wise deletion which removes data cases containing missing values in any of the variables involved in the current CI test. The pseudo-code of GS (Margaritis) (003) is presented in Algorithm [] Note that we slightly modify the Grow phase, i.e., line 4-6, of GS to eliminate dependency on the order of the variables in the data (Kitson & Constantinou, 2022).
Given the faithfulness condition, Assumption and Assumption rectness of GS with test-wise deletion.
2 Proposition D describes the cor-
Proposition 1. Given the faithfulness condition and Assumptions and for any observed variable V; a m-graph G, the output of GS(V,, v V R (V., R.). D) is the MB (V;) in G.
The proof of Proposition is provided in Appendix A Therefore, an intuitive way determine the relevant features for given variable is to apply the function GS(V. V U Vm u R (V., R.) D) on every V; E Vmm However, this is impractical since the maximum size of the conditioning sets used for CI testing is V + 2 -3. In practice, the accuracy of CI tests drops dramatically as we increase the size of the conditional set (Tsamardinos et al. 2003). To address this, we propose the Markov Blanket Feature Selection (MBFS, Algorithm 2 that aims to restrict the maximum size of the conditional set used by CI tests to | |
MBFS involves two phases, where the first phase involves learning the intrinsic MB of each partially observed variable. Given a m- m--raph G. we define the intrinsic MB of a variable V; as the set of variables that are still in the MB of V, after removing all indicator variables from G. We denote the intrinsic MB of V; by iMB (V.). Note that iMB (V;) is not necessarily equivalent to the set of observed variables in MB (V.). This is because the missing indicators might be a common effect of two observed variables. For example, the intrinsic MB of Vi in Figure Ic is {V2, V3, V4, Vs),
'oo impute the missing values of an incomplete variable, we consider its MB, rather than only its parent variables, for two reasons. Firstly, the parents of an incomplete (or even a complete) variable are not guaranteed to he identifiable from observational data. Secondly, the MB contains the set of nodes that can make the given variable independent over all other variables present in the input data.
4

---

Published as a conference paper at ICLR 2023
Algorithm
The Grow and Shrink (GS) algorithm with test-wise deletion Input target variable candidate variables S, data D Output candidate Markov Blanket CMB of X
1:
procedure GS(X,
2: CMB <
3: repeat
Grow phase
4:
5: any X ff. Se CMB, R = for S; E S then add S; with the lowest p-value to CMB remove S; from S
6:
7:
end if
8:
until CMB stays unchanged
9:
for each Y CMB do
D Shrink phase
10:
if TT Y CMB {Y}. R((jUCM = 0 then remove Y from CMB
12:
end if end for
13:
14: return CMB
15:
end procedure
whereas the standard MB would have also included Ve and R3. is worth noting that, during phase 1, some nodes that do not belong in iMB (V;) may still be included in the output CMB. However, as we show in Appendix B these nodes are still in the MB (V,) in m-graph. The phase aims to learn all
the parents of the missing indicators, in order to complete the feature set of MB (V)). Proposition 2
states that MBFS is capable of learning MB (V:) from missing data for any V E V in a m-graph and thus, it could serve effective feature selection approach for imputation algorithms.
Algorithm 2 Markov Blanket- based Feature Selection (MBFS)
Input partially observed variable V,, data D Output candidate Markov Blanket CMB of V;
1:
procedure MBFS(V,, D)
Phase (discover intrinsic MB)
2: CMB < GS (V, V U Vm (V). D)
Phase (discover other variables MB caused by indicators)
3: for each R, R [Ra} do
4: CPS < Vo U V {VG
5:
for each Vx E CPS do
6:
if R, 1L V& S. R,R 0 for any S C CPS then
7:
remove VE from CPS end for
8:
end if
9:
10:
if V, E CPS then
11:
CMB +- CMB U {Ry} CPS
12:
end if 13: end for
14: return CMB
15:
end procedure
Proposition 2. Given the faithfulness condition, Assumptions 7 and for any observed variable V, in m-graph G, MBFS(V,, D) returns MB(V.) in g.
The proof of Proposition is provided Appendix Appendix C discusses the implications on the learning performance of MBFS when the Assumptions and are violated. We then propose modified version of FossForest that incorporates MBFS as feature selection process. The modified version of MissForest, which call Markov Blanket MissForest (MBMF), takes the feature set MBFS (V,, D) for each partially observed variables V,, as opposed to considering all the other observed variables the explanatory features of V; in the Random Forest regression model used in
5

---

Published as a conference paper at ICLR 2023
MissForest. In other words, MBMF accounts for the possible causal relationships between partially observed variables and the missing indicators, to minimise the risk considering irrelevant observed variables for imputation by MissForest.
4 EXPERIMENTS
We test the proposed MBMF algorithm with reference to the standard version of MissForest (MF), the commonly used imputation algorithms Mean and Mode, the K-Nearest Neighbour (KNN), and two state-of-the-art algortihms; the softlmpute and GAIN algorithms. While the evaluation includes experiments on both continuous and categorical data, some of the other algorithms can only pro- cess one of the two types of imput data and hence, their application restricted to continuous data (Mean and GAIN) or categorical data (Mode). We use the scikit-learn python package (Pedregosal al. 2011 I) to test the Mean, Mode and KNNN algorithms, the MissForest R package (Stekhoven & Stekhoven 2013) to test MF, the softImpute R package (Hastie et al.) 2015) to test SoftImpute, and the publicly available source code of GAIN. The implementation of MBME, described in this paper,
is available at: http:///tituu..oom///eeeeeoiiiie
Blaeer--Bannnntt-eeeeeeee Bew.aaaeeeeaseeeeeetteeeeeeeeeee Feature-Selection
MBMF is applied to continuous data using the Pearson's 's correlation test for CI tests, and to categor- ical data using the G-test statistic, both of which are the default choices for GS. We also consider the default threshold for independence, which is 0.1 for CI p-value tests. The other algorithms are also tested with their default dyper-parameten implemented in their corresponding packages listed above.
4.1 SYNTHETIC CASE STUDIES BASED ON REAL-WORLD. BNS
We first evaluate the algorithms by applying them synthetic data sampled from three BNs, ECOLI70, MAGIC-IRRI and ARTH150, taken from the bulearn repository (Scutari 2010). De- tails about these graphical networks for each network with sample sizes 500, 1000, 2000 and 3000. Then, for each complete data set, create nine incomplete data sets composed of different combinations of missingness rates (i.e., 10%, 30% and 50%) and missingness assumptions (i.e., MCAR, MAR and MNAR). Appendix describes the process followed generate data sets with different types of missingness.
be found in Appendix D We generate complete data sets
4.2 EVALUATION PROCESS
The imputation accuracy is evaluated using two different approaches. The first approach involves retrieving the Root Mean Squared Error (RMSE) between imputed data and complete data. Because RMSE is sensitive to the discrepancy between absolute data values, we normalise the complete data column-wise and re-scale the imputed data with the same normalisation parameters to eliminate bias.
The second approach involves assessing the impact of imputation on structural learning accuracy. We do this by comparing the Completed Partially Directed Acyclic Graphs (CPDAGs) learned by the state-of-the-art GES causal structure learning algorithm (Chickering 2002) from imputed data sets produced by the different imputation algorithms. The second approach is helpful because, while it reasonable to assume that higher imputation accuracy helps causal machine learning, is possible that some imputed values are more important than others. Causal structure learning represents good approach test this, and we use the Fl score to measure the accuracy of graphical structures learned by GES, as follows:
2TP
F1
2TP + FP FN'
where TP, FP and FN represent the number of true positive, false positive and false negative edges in the learned CPDAG, relative to the true CPDAG. For more information how to retrieve the CPDAG DAG, please refer to (Chickering 2002). Readers are also referred to (Kitson et al, 2023) for review of structure learning.

---

Published as a conference paper at ICLR 2023
MCARR
MAR
MNAAR
PACE SUAL DuscRu
0.12
0.10
0.21
0.18
"NGE BuISRN
BBM
0.12
SAIN
Softmpute
0.200
NN05 Busan
0.375
0.150
0.325
1000
2000
3000
1000
2000
3000
1000
2000
2000
Data size
Figure 2: Average RMSE between complete and imputed data produced by the different algorithms. A Lower score represents better performance.
4.3 RESULTS
Figure 2 depicts the average RMSE of imputed data produced by the different algorithms under different sample sizes. Note that the results we report on GAIN are, to some degree, inconsistent with the results presented in the original paper YYoon al. 20088) but are consistent with the results presented in follow-up studies (You et al., [2200] Nazabal et al. 2020 Kyono et al.) 2021 Jarrett al.] 2022). In general, the proposed MBMF algorithm is found to outperform the baseline MF under all scenarios of missingness. Specifically, for MCAR and MAR scenarios with missing rate 10%, MBMF provides a considerable improvement over MF, but this improvement diminishes with the higher missing rates of 30% and 50%. This is because higher missing rate tends to decrease the sample size the test-wise deleted data, which in turn reduces the accuracy of CI tests and the discovered MB set by MBFS. Importantly, MBMF outperforms MF considerably under all MNAR settings, which better reflect real-world missingness that is generally systematic. None of the other imputation algorithms provide satisfactory performance in terms of RMSE; at least relative to the MBMF and MF algorithms.
Figure 3 presents the average F1 scores corresponding the graphs learned by GES, given the imputed data sets produced by the different imputation algorithms, and across the different sample sizes. While this evaluation approach decreases the discrepancy in performance scores between the top performing imputation algorithms,, the results are consistent with those presented in Figure 2 since MBMF and MF are found to perform better than the other algorithms in almost all cases, and MBNF performs better than MF in most experiments. Specifically, MBMF and MF produce similar performance when the rate of missingness is lowest 10%, with their performance being close to that produced with complete data (dashed line). These results serve as empirical evidence that both MFBF and MF imputation algorithms perform exceptionally well with relatively low rates of missingness. When the rate of missingness increases to 30%, MBMF performs better than MF in most cases. However, when the rate missingness is highest, at 50%, there is no clear winner between MBMF and MF.
Lastly, we evaluate the computational efficiency of MBMF relative to the original MF. As shown in Figure MBMF is generally more efficient than MF. Note that while MBMF involves additional phase needed to perform feature selection, the additional time spent by MBMF in that phase is countered by the reduced time MBMF spends actually impute values. This is because MBMF

---

Published as a conference paper at ICLR 2023
MCAR
MAR
MINAR
0.7
0.5
NAOT BascRu
0.5
0.7
MF
BMF
"NDE BuISRN
M251
GAIN
mptte
0.3
Complete
NN05 aAU BusaN
0.5
0.3
2000
2000
3000
1000
2000
3000
1000
2002
3000
Data size
Figure 3: Average Fl scores of the graphs learned by GES from data imputed by the different algorithms. A higher Fl score represents better performance. The dashed line represents the perfor- mance of GES when applied complete data.
sélection prase
Impulation phass
300
Exocutiio time secands
350
200
230
520
1032
2000
3000
MC4R
MNKA
10%
10%
50%
Data size
Missing mechanism
Missing rate
Figure 4: Average execution time MF and MBMF under different sample sizes, mechanisms of missingness, and rates of missingness.
will almost always consider less features than MF during the imputation phase. Specifically, MBMF slower than MF at the lowest sample size, becomes increasingly faster than MF with increasing sample size. Averaging the results across the different mechanisms of missingness and missing rates also shows that MBMF is, in general, more efficient than MF. Note that MF slightly more efficient when the rate of missingness is at its highest, 50%, since MF trains its RF regression model on observed data rows only; implying that a higher rate of missingness decreases the training data passed to the RF. The impact of the rate of missingness is higher MF than MBMF since the number of independent features considered by MF is, in general, considerably higher than those considered by MBME.
4.4 REAL-WORLD CASE STUDY
We repeat the evaluation by applying the imputation algorithms to six real-world data sets retrieved from the UCI data repository (Dua & Graff (017)). A summary of these data sets is given in Table in Appendix F We simulate missingness using the same strategy described in Section 4.1 Specif-
8

---

Published as a conference paper at ICLR 2023
Mean/Mode
KNN
SoftImpute
GAIN
MF
MBMF
Iris
.2842.085
.103:.029
.198:.055
.164=.00
.092 = .026
.092+.025
Breast
.149+.012
.113+.009
.140+.040
.098=.029
.057 + .009
.054..009
Wine
.123+-009
.109+.003
.101+.010
.106=.009
.039
.40
Game
5563..020
.501+.072
.005 = .064
.4961.070
Car
.675+.074
.613#.073
.655 = .107
.593+.061
Mushroom
.433..022
.294..083
207 = .060
.203+.058
Table 1: Average RMSE and PFC scores, and their standard deviations, for different imputation
algorithms and real-world data combinations. Lower RMSE and PFC scores represent better perfor- mance.
ically, for each complete real data set, we generate nine incomplete data sets composed of different rates and assumptions of missingness.
We use RMSE to evaluate the imputation accuracy when the algorithms are applied to continuous data, and the Proportion of Falsely Classified entries (PFC) when applied categorical data. Note that the second evaluation approach, which involves investigating the impact of imputation accuracy on causal structure learning as described in subsection 4.2, is unsuitable here since we do not have ground truth causal graph.
Table presents the average scores and standard deviation. The results show that MBMF and MF continue to outperform the other algorithms, and that MBMF continues outperform MF in most of the experiments. On this basis, we conclude that the results obtained from the woal-world data sets consistent with the results obtained from the synthetic data sets.
5 CONCLUSION
This paper described novel feature selection algorithm, called MBFS, that recovers the Markov Blanket of partially observed variables based on the graphical expression of missingness known as the m-graph, which captures observed variables together with missingness indicators and the possible causal links between them. We then incorporated MBFS into the imputation process of MissForest, to formulate new algorithm suitable for imputation under both random and systematic missingness, which we call MBMF
Empirical experiments based on both synthetic and real-world data sets show that MBMF outper- forms the baseline MissForest in most of the experimental settings, and outperforms considerably other well-known state-of-the of-tth-art imputation algorithms under both random and systematic miss- ingness. Moreover, while MBMF incorporates an additional learning phase needed to perform fea- ture selection for each partially observed variable found in the input data, the results show that MBMF is generally more efficient than the baseline MissForest, especially at larger sample sizes where efficiency matters the most. This is because the time saved during imputation due to prior feature selection is higher than the additional time spent determining the best features for each par- tially observed variable.
Because the feature selection phase can be independent of the imputation phase (i.e., by using the MBFS algorithm alone), future research works could extend this work to different relevant direc- tions where feature selection is deemed to be important. For example. MBFS could be combined with other imputation algorithms, including those based on deep learning (atttei & Frellsen 2019, Fortuin et al. 2020 Lin et al., (202)) which are generally powerful but which tend to be time con- suming and to overfit the data, since they typically process large numbers of uncorrelated features. Moreover, since each MB discovered by MBFS could be used to construct the complete m-graph of the input data set, a rather different possible direction for future research would be to investigate the capability of MBFS in recovering the entire m-graph of the input data. This task would require structural rules to deal with collisions between MBs as well as cycles, and would essentially convert MBFS into structure learning algorithm.

---

Published as a conference paper at ICLR 2023
REFERENCES
Ahmad R Alsaber, Jiazhu Pan, and Adeeba Al-Hurban. Handling complex missing data using ran- dom forest approach for an air quality monitoring dataset: case study of kuwait environmental data (2012 to 2018). International Journal of Environmental Research and Public Health, 18(3): 1333, 2021.
Peter C Austin, lan R White, Douglas S Lee, and Stef van Buuren. Missing data in clinical research: tutorial on multiple imputation. Canadian Journal of Cardiology, 37(9):332--133. 2021.
Shatha Awawdeh, Hossam Faris, and Hazem Hiary. Evoimputer: An evolutionary approach for missing data imputation and feature selection in the context of supervised learning. Knowledge- Based Systems, 236: 107734, 2022.
Amanda N Baraldi and Craig K Enders. An introduction to modern missing data analyses. Journal of school psychology, (88(11553,,, 2010.
Fanyu Bu, Zhikui Chen, Qingchen Zhang, and Laurence T Yang. Incomplete nigh-dimensional data imputation algorithm using feature selection and clustering analysis on cloud. The Journal of Supercomputing, 72(8):9977-299) 2016.
David Maxwell Chickering. Optimal structure identification with greedy search. Journal of machine learning research, 3 3(Nov):007-554 2002.
Gauthier Doquire and Michel Verleysen. Feature selection with missing data using mutual informa- tion estimators. Neurocomputing, 90:3-11, 2012.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL Bttthtttp//cchi//
ics. uci edu/ml
Mohamad Faiz Dzulkalnine and Roselina Sallehuddin. Missing data imputation with fuzzy feature selection for diabetes dataset. SN Applied Sciences, 1(4):1-12, 2019.
Vincent Fortuin, Dmitry Baranchuk, Gunnar Rätsch, and Stephan Mandt. Gp-vae: Deep probabilis- tic time series imputation. In International conference artificial intelligence and statistics, pp- 1651-1661. PMLR, 2020.
Trevor Hastie, Rahul Mazumder, Jason D Lee, and Reza Zadeh. Matrix completion and low-rank svd via fast alternating least squares. The Journal of Machine Learning Research, 16(1):336733402, 2015.
Minh Hieu Nguyen, Jimmy Armoogum, and Emeli Adell. Feature selection for enhancing purpose imputation using global positioning system data without geographic information system data. Transportation Research Record, 2675(5):75-87, 2021.
James Honaker, Gary King, and Matthew Blackwell. Amelia A program for missing data. Journal of statistical software, 45:1-47, 2011.
Daniel Jarrett, Bogdan C Cebere, Tennison Liu, Alicia Curth, and Mihaela van der Schaar. Hy- perimpute: Generalized iterative imputation with automatic model selection. In International Conference on Machine Learning, pp. 9916-9937. PMLR, 2022.
Heejin Jin, Surin Jung, and Sungho Won. missforest with feature selection using binary particle swarm optimization improves the imputation accuracy of continuous data. Genes Genomics, (4(6))651-655) 2022.
Neville K Kitson and Anthony C Constantinou. The of orderin bayesian network structure learning, 2022. URL
08952
Neville Kenneth Kitson, Anthony C Constantinou, Zhigao Guo, Yang Liu, and Kiattikun Chobtham. A survey of bayesian network structure learning. Artificial Intelligence Review, pp. 1-94, 2023.
Trent Kyono, Yao Zhang, Alexis Bellot, and Mihaela van der Schaar. Miracle: Causally-ane aware imputation via learning missing data mechanisms. Advances in Neural Information Processing Systems, 14:23806-2388 2021.
10

---

Published as a conference paper at ICLR 2023
Katherine Lee and John B Carlin. Multiple imputation for missing data: fully conditional specifica- tion versus multivariate normal imputation. American journal of epidemiology, 171(5):624-00 632, 2010.
Wei-Chao Lin and Chih-Fong Tsai. Missing value imputation: review and analysis of the literature (2006-2017), Artificial Intelligence Review, 53(2):1487-1509 2020.
Wei-Chao Lin, Chih-Fong Tsai, and Jia Rong Zhong. Deep learning for missing value imputation of continuous data and the effect of data discretization. Knowledge-Based Systems, 239: 108079, 2022.
Roderick Little, Ralph D' Agostino, Michael Cohen, Kay Dickersin, Scott S Emerson, John Farrar, Constantine Frangakis, Joseph W Hogan, Geert Molenherghs, Susan A Murphy, al. The prevention and treatment of missing data in clinical trials. New England Journal of Medicine, 367
(14):1355-1366) 13600 2012.
Chia-Hui Liu, Chih-Fong Tsai, Kuen-Liang Sue, and Min-Wei Huang. The feature selection effect on missing value imputation of medical datasets. applied sciences, 10(7):2344, 2020.
Dimitris Margaritis. Learning bayesian network model structure from data. Technical report, Carnegie-Mellon Univ Pittsburgh Pa School of Computer Science, 2003.
Pierre-Alexandre Matei and Jes Frellsen. Míwae: Deep generative modelling and imputation of incomplete data sets. In International conference on machine learning, pp. 4413-441 PMLR, 2019.
Karthika Mohan, Judea Pearl, and Jin Tian. Graphical models for inference with missing data. Advances in neural information processing systems, 26, 2013.
Boris Muzellee, Julie Josse, Claire Boyer, and Marco Cuturi. Missing data imputation using optimal transport. In International Conference Machine Learming, pp. 7130-7140. PMLR, 2020.
Alfredo Nazabal, Pablo M Olmos, Zoubin Ghahramani, and Isabel Valera. Handling incomplete heterogeneous data using vaes. Pattern Recognition, 107:107501, 2020.
Judea Pearl. Probabilistic reasoning in intelligent systems: networks of plausible inference. Morgan kaufmann, 1988
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Ben Omega Petrazzini, Hugo Naya, Fernando Jopez-Bello, Gustavo Vazquez, and Lucfa Spangen- berg. Evaluation of different approaches for missing data imputation features associated to genomic data. BioData mining, 14(1):1-13 2021.
Donald B Rubin. Inference and missing data. Biometrika, 63(3):881-59,, 1976.
M Scutari. Learning bayesian networks with the bulearn package. Journal of Statistical Software, 35(3), 2010.
Amir Masoud Sefidian and Negin Daneshpour. Missing value imputation using novel grey based fuzzy c-means, mutual information based feature selection, and regression model. Expert Systems with Applications, 1 115:68-94 94,899944758 2019.
Dušan Sovilj, Emil Eirola, Yoan Miche, Kaj-Mikael Björk, Rui Nian, Anton Akusok, and Amaury Lendasse. Extreme learning machine for missing data using multiple imputations. Neurocomput- ing, 174:220-231, 2016.
Daniel Stekhoven and Peter Bühlmann. Missforest- -ar-pammetri missing value imputation for mixed-type data. Bioinformatics, 28(1):1 12-118, 2012.
Daniel Stekhoven and Maintainer Daniel J Stekhoven. Package 'missforest'. R package version, 1, 2013.
11

---

Published as a conference paper at ICLR 2023
Cao Truong Tran, Mengjie Zhang. Peter Andreae, Bing Xue, and Lam Thu Bui. Improving perfor- mance classification on incomplete data using feature selection and clustering. Applied Soft Computing, 73:848 861, 2018.
loannis Tsamardinos, Constantin F Aliferis, Alexander R Statnikov, and Er Statnikov. Algorithms for large scale markov blanket discovery. In FLAIRS conference, volume 2, pp. 376-380. St. Augustine, FI., 2003.
Stef Van Buuren and Karin tions in r. Journal of statistical software, 45:1-67, 2011.
mice: Multivariate imputation by chained equa-
Leland Wilkinson. Statistical methods in psychology journals: Guidelines and explanations. Amer- ican psychologist, 54(8):594, 1999.
Jinsung Yoon, James Jordon, and Mihacla Schaar. Gain: Missing data imputation using generative adversarial nets. In International conference on machine learning, pp. 5689-5698. PMLR, 2018.
Jiaxuan You, Xiaobai Ma. Yi Ding, Mykel J Kochenderfer, and Jure Leskovec. Handling missing data with graph representation learning. Advances in Neural Information Processing Systems, 33: 19075-19087, 2020.
Kui Yu, Yajing Yang, and Wei Ding. Causal feature selection with missing data. ACM Trans. Knowl. Discov. Data, 16(4), jan 2022. ISSN 1556-4681, doi: 10.1145/3488055, URL https: 7/ //doi. wwoiioorg/10..
Shichao Zhang. Nearest neighbor selection for iteratively knn imputation. Journal of Systems and Software, (5(11):251--255, 2012.
Yifan Zhang and Peter J Thorburn. Handling missing data in near real-time environmental moni- toring: system and a review of selected methods. Future Generation Computer Systems, 128: 63-72, 2022.
A PROOF OF PROPOSITION
Given Assumption T andd Assumption 2 R2 is child of V; nor parent of any other variables. Therefore, we do not need to consider R; when learning MB (V,).
We first show that the output Candidate Markov Blanket (CMB; ie., output of Algorithm set learned at the Grow phase contains all the variables of MB (V)). Assume Y is a parent or child of V,, given the faithfulness condition, V, J.. Y CMB, RIv.YYJCME
0 for any CMB C V V U R (V,, R. Therefore, Y will always be added to CMB during the Grow phase. Assume Z + Y and that Y is also child of V,, Once Y is added to CMB, V, and Z will not be d-separated by CMB U RUV.Y) and thus, Z will also be added to CMB.
Lastly, show that the Shrink phase preserves the variables in MB (V:) only. Let assume that is the first variable in MB (V+) when the algorithm enters the Shrink phase and attempts remove variables in CMB, which have already been added during the Grow phase. Because MB (V4) - (T} C CMB, irrespective of T being a neighbour or parent of a child of V,, we always have V ML T (M)-(T) [(V.jUCMR = 0 given the faithfulness condition. Therefore, no variable in MB (V)) will be removed during the Shrink phase. On the other hand, if we assume T & MB (V)), since MB (V,) C CMB U Riv.JucMr JUCMB - { we have V; IL T CMB {T') Rey.jucME = 0 given the faithfulness condition and thus, I" will be removed from CMB.
B PROOF OF PROPOSITION 2
For convenience, in this section we denote path p as the path being blocked by a variable set S if there is at least one node p that satisfies either 1) it is non-collider and in S 2) it is collider and neither it nor any of its descendants are in S. Thus, if all paths between V; and V; blocked by S, V, and Vy d-separated by S. We firstly show that the CMB returned by
12

---

Published as a conference paper at ICLR 2023
-[V),, (phase of MBFS) contains all but only variables in iMB (V)), in addition to some other observed variables belonging to MB (V.).
GS (V, V U V
For the Grow phase in GS (V, V U UV {VV) D), assumee that Y is parent child of Vi, and that given the faithfulness condition, Vi H Y vo U Vm - {V.}. Therefore, Y will be added in CMB. Assume that Z is a parent of an observed variable Y, which is a child of V,, After Y is added to CMB, Vi and Z can no longer be d-separated by CMB U (V; (V.YY)CMM) Thus, Z will also be added to CMB eventually. As a result, all variables in iMB (Vi) will be included in CMB at the end of the Grow phase.
R(V, JUCMB = for any CMB S
For the Shrink phase, we prove two things: 1) that all the variables in iMB (V:) will remain in CMB, and 2) that all the variables not in the MB (V;) will be removed from CMB.
1. Suppose that I' is the first variable in iMB (V:) that the algorithm attempts to remove from CMB, and that V; and T are connected by either direct edge or a path through a collider in iMB (V)). Therefore, T and V,, cannot be d-separated by CMB U (va)UCM - {T), since iMB (V)) S CMBUR www...u..--e..... (T). Then, we have Vg J.. T CMB (T). RAv.JUCMB = 0 given the faithfulness condition.
2. Suppose that T is an observed variable in CMB such that T & MB (V.). Given the faith- fulness condition, all we need to prove is that all paths between Vi and T are blocked by CMB R(v.)ucm- {T}.
For path p between V, and T composed by observed variables only, there is least one non-collider node on that belongs to iMB (Vi), such that is blocked by CMB U Riv.juCMB { since iMB (V,) S. CMB U RCC {T').
For path p between VÚ2 and T contains indicator variables, when an observed node V, is adjacent to Vú on p. at least one non-collider (either V, or the parent of Vÿ) on p will also be in iMB (V,). This is because no indicator variable can be a parent of V according to Assumption 1 which implies that p will be blocked by iMB (V,) as well as CMB U R.v.).CMB
{T}. If the adjacent node of V42 on p an indicator variable Ry, then according to Assumptions and 2 Rg * R, and it must have another parent V& + T on p since T & MB (V.J. If K & RCMB. path p is blocked by CMB U R (V,,JUMM - [T]. If Rí E ReMB, V must be included in CMB during the Grow phase, and cannot be removed during the Shrink phase since VÚ2 and Va cannot be d-separated by CMB U Riv, REVCME Therefore, p is still blocked by CMB U RVV,Jucma - {T} since Vi2 is non-collider on in CMB U RVV:)UCMB - (T).
In the phase 2 of MBFS, all the parents of each R2 should remain in CPS they cannot be d- separated from R2 given any set composed of observed variables. Besides, all of the other variables should be removed from CPS they can be d d-separated from R, given any set that contains all the parents of Rg. As a result, CMB should return MB (V)) at the end of phase 2.
C THE IMPLICATIONS OF VIOLATING ASSUMPTIONS AND 2 OF MBFS
If the Assumption is violated (e.g., when an indicator variable is also a parent of some other variable), then both phases of MBFS may produce redundant variables in the CMB set. Let us suppose that the true m-graph the one shown in Figure If we use MBFS to learn the MB of Vi. then V/2 will be included in the CMB set of Phase 1, and this is because V1 and V/2 are independent only when conditioning on R4, whereas V4 and V1 are always independent. This implies that MBFS will never test V1 and V22 for CI conditional in the CPS set of Rs and thus, it will also be added in CMB of Vı.
R4 during Phase 1. In Phase 2, Va will be included
If Assumption 2 is violated (c.g., when partially observed variable V, is also the parent of its own missingness indicator R2), this will cause MBFS not to include R, in the CMB of V,, since MBFS would not consider Vú as candidate variable in the CPS of R2 during Phase 2.
13

---

Published as a conference paper at ICLR 2023
R
V4
O
O
R4
V3
2
VI
VE
VE
R
Figure 5: A hypothetical m-graph used to described the implications of violating Assumption | of MBFS. Shaded nodes represent partially observed variables.
D SUMMARY OF THE REAL-WORLD BNS IN SYNTHETIC EXPERIMENTS
Number of variables
Number of edges
Data type
ECOLI70
46
70
Continuous Continuous
MAGIC-IRRI
64
102
ARTH150
107
150
Continuous
Table 2: Summary of the real-world BNs used in Section 4.1
E MISSING VALUE GENERATION PROCESS
We randomly choose 50% of the variables to be made partially observed. We use to represent the rate of missingness for each of those variables. This process differs depending on the underlying assumption of missingness. Specifically,
1. For the MCAR condition, value in V; is removed with probability P PRR:- = a.
2. For the MAR condition, a fully observed variable V, E Vo is randomly assigned as the parent of R,, For continuous data, we denote the highest 10th-quantile of V2 as qj. and remove values in V/2 with the following conditional probabilities: P P(R, = = V > qg) =
2a, (R, = 11 V2 <= 4n) = $a. For categorical data, we select a set of states ey of V, such that < (V, v,) < 1, then remove values in V, with the following
conditional probabilities: P P(R,- = 1 V, + UÚ) = 2a, P = 1 V, =,) 2a
3. For the MNAR condition, partially observed variable V, Vm [V}} is randomly assigned as the parent of R, Variable values are then removed for each Vi using the same strategy in the case of MAR.
F SUMMARY OF THE REAL- WORLD DATA SETS
Number of variables
Number of instances
Data type
Iris
4
150
Continuous
Breast
30
569
Continuous
Wine
11
1599
Continuous
Game
10
958
Categorical
Car
7
1728
Categorical
Mushroom
22
8124
Categorical
Table 3: Summary of the real-world data sets used in Section 4.4
14


Reviews:
[{"review_id": "KovNVFkaiYv", "reviewer": "Reviewer_p56x", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a feature selection method based on markov blanket under missing data regime and apply missforest after feature selection. The main contribution is the feature selection algorithm.", "summary_of_the_review": "I think this paper is at the borderline. The authors do not compare to many new SOTA methods, which I believe have better results than MissForest. But it can be the reason that MissForest is not that good so MBMF is not as good as those SOTA methods. ", "strength_and_weaknesses": "Strength:\n1. The authors propose a markov blanked based feature selection algorithm for missing data. The selected features are then used in MissForest. \n\nWeakness:\n1. The authors should include more recent works to claim that their method is SOTA. MissForest is kind of old (10 years old). More recent SOTA should be included. For example, \nMIRACLE (Kyono et al., 2021)\nSinkhorn (Muzellec et al., 2020)\nSoftImpute (Hastie et al., 2015)\nHyperImpute (Jarrettet al., 2022)\n2. MimMB is similar to MBMF. Empirical comparison should also be included.\n3. Unlike MimMB, the authors claim that improves computational efficiency. A comparison of execution time is more persuasive. If it's trivial, the authors can correct me without making this new result of computational efficiency. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "K4apls54ZU", "reviewer": "Reviewer_X4h7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose MBMF, essentially a feature selection procedure for a second step imputation method. With MBMF the authors aim to recover the m-graph that governs a system's missingness (in the tabular domain). Using the m-graph, the authors recover a variable of interest's markov blanket which are used as features for imputation. The second step uses the selected features to impute a missing variable. \n\nTheir experiments show that MBMF combined with MissForest outperforms the most adopted imputation strategies. ", "summary_of_the_review": "I think the paper is well written, interesting, and touches on some interesting ideas to marry causal discovery and imputation. I was left with a few questions I hope the authors can answer in their rebuttal.", "strength_and_weaknesses": "STRENGTHS\n\n* Clearly, causality and structure are important elements in imputation, particularly in the tabular domain. Imputation is an important research area that should receive more attention than it currently does.\n\n* I found the paper quite well written. Also, I find it interesting that the authors base their method on a tried and tested algorithm from 2003. \n\nWEAKNESSES\n\nI think the paper was interesting and have only a few questions I would like the authors' opinion on.\n\n* What exactly is the motivation for using the Markov blanket instead of the parents of X_i? Kyono et al. (2020) seem to be doing well with the covariate's parents? (Note that an m-graph is _causal_, saying it is recovered allows assuming that the parents of a variable are _actually_ the causes). \n\n* Would it be useful to try MBMF with other algorithms (beyond MissForest)? If so, why haven't you done this and could you? If not, why not?\n\n* It seems GAIN is not performing very well, despite it being considered the sota, could you elaborate or reason why this is the case?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6xnHY1f-B8", "reviewer": "Reviewer_HRJ6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This submission proposes a new imputation algorithm called Markov-Blanket Miss-Forest (MBMF), which involves two phases: Markov Blanket-based Feature Selection (MBFS), and MissForest (MF) imputation (Stekhoven & Buhlmann, 2012). \n\nFor each partially observed variable V_i, MBFS finds its Markov Blanket within some m-graph which is referred to as the graphical expression of missingness proposed by Mohan et al. (2013).  Then, elements of the Markov Blanket are considered as explanatory features of V_i in the Random Forest regression model used in MF.  \n\nSome experiments are conducted to assess the usefulness of MBMF, in comparison with the original MF, and simpler imputation algorithms (Mean, Mode and KNN algorithms).", "summary_of_the_review": "I think \"MBFS\" & \"imputation algorithm\" is an interesting imputation approach and can be implemented with different imputation algorithms. I gave a few questions in \"Strength And Weaknesses\".  ", "strength_and_weaknesses": "Strengths\n- Markov Blanket-based Feature Selection (MBFS) and imputation is interesting\n- Experimental results show that MBMF can provide promising imputation accuracy  \n\nWeaknesses: I think the paper is interesting and I have a few questions. \n- Assumption: Is the assumption 1 \"R_i could only be a leaf node in m-graph\" made to make the problem of finding Markov Blankets easier? Or is it a theoretical guarantee?  \n- M-graphs: For each data set, can MBFS recover the m-graph which informs the missingness mechanisms? For each given incomplete data set, can the output of MBFS suggest which underlying assumption of missingness was used to create that incomplete data set?   \n- Scalability: Experiments are done on data sets with small and moderate sizes (at most 30 variables and at most 8124). Can you comment on the scalability of MBFS?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6jD3IaAItmY", "reviewer": "Reviewer_14vZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method for feature selection to be used prior to estimating an imputation model. The method is based on finding the Markov blanket for each partially observed variable. In synthetic and semi-synthetic experiments, the methods slightly outperforms an imputation model trained without feature selection with a lower total run time (feature selection + model estimation).", "summary_of_the_review": "Overall, I found this paper exceptionally clear and well-written. The method is interesting and the experiments are reasonable and well-executed. My only overall concern is regards motivation. By and large, ML methods favor general regularization over using feature selection as a preprocessing step, so I was a bit unconvinced by the initial motivation that feature selection is necessary to improve imputation performance. I think this intuition is born out by the experiments which seem to demonstrate only a small improvement over the base method applied without feature selection. I think the intro would be improved with a bit more justification for the approach and, specifically, why the authors believe feature selection is the right approach here when regularization has worked so well in other settings. With that said, I still think the approach is reasonable and merits publication.\n\nMinor comments:\n1. Overall, I thought the background provided in the intro was excellent. My only quibble is that, in my experience, the most commonly used approach to missing data in statistical settings is multiple imputation and it might be worth mentioning this approach in the intro. Note that improved imputation models mean improve MI.\n\n2. Intro: Modem --> Mode\n\n3. Figures 1 and 2: I think these figures could be cleaned up by organizing them in a 3x3 grid. If you are concerned about space, I recommend sharing x and y axes within the grid.", "strength_and_weaknesses": "See below.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster