Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a new method for refining existing baseline missing data imputation algorithms. The method works by starting from an initial estimate from another baseline algorithm, then pass this estimate to two neural networks for refined estimation. The two neural networks are regularized based on intuitions from causal inference, such that the causal missingness of the data can be captured. The refinement process is iteratively run until convergence. The contribution of the paper is a new method for missing data imputation inspired by causal inference concepts.
main_review: + The proposed method is inspired by causal inference, and this is less explored in the literature. Successful application of causal inference to missing data would be a significant step.

+ The paper ran extensive experiments and overall the benefits of the algorithm is demonstrated (see some minor points below)

+ The paper is well organized and easy to follow.

- I am a little bit concerned about the identifiability problem. As the paper mentioned in Line 205 itself, "Discovering a causal graph requires complete data. However, this is not the case for missing data." From my understanding, identifying causal structures requires access to interventions data, and I am not sure how the proposed approach connects to the identifiability problem, and also, how the four assumptions help define the identifiability. For example, is any theorems from [18] that could help explain why the proposed method can help recover the missing data/ missing mechanism? I think the paper lacks detailed discussion on this part.

- Following the previous point, if we are trying to recover the causal structures, is it possible to have some simulation studies to understand how well we are approximating the underlying causal graph? I think Figure 2 is trying to show the adjacency graph is improved over the course of the training. When we are synthesizing the missingness of the data, it appears possible to report how well we are learning the underlying graph by using the matrix B and compare to our synthesis.

- (Minor) It would be helpful for readers if the three missing data mechanisms are explicitly discussed.

- (Minor) The paper discussed about robust optimization in the introduction, but I think this discussion is forgotten in the later sections. How does the proposed approach connects to robust optimization is an interesting topic.

- For the experiments part, some of the results have really large error bars, for example, Figure 10. b), Figure 12 [life expectancy, community, yeast] in Appendix, and looks like the results would not be significant if you compare the baseline vs baseline + MIRACLE. I am wondering whether this provides any insight on the proposed algorithm

[18] Karthika Mohan, Judea Pearl, and Jin Tian. Graphical models for inference with missing data. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26, pages 1277–1285. Curran Associates, Inc., 2013.
limitations_and_societal_impact: - On the computational complexity, I think the paper doesn't address it well. There are some discussions in Appendix E., "Computational time scales linearly with increasing the number of input samples.". However, a more important aspect, from my point of view, is the complexity w.r.t to the feature size. Since a matrix exponential is used in 3.3 Causal regularizer, the complexity is $O(d^3)$, and it would probably be very challenging to apply in a large scale applications.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a new framework to impute missing values while preserving the causal structure of the data. The algorithm works by refining the imputations of a baseline method and simultaneously learning the missingness generating mechanisms (which cover different scenarios such as MCAR, MAR, MNAR). The proposed method obtains improved results when compared to the baselines in both synthetic and real experiments.
main_review: In general, I think this is a good paper. It addresses a very timely problem of great interest to the machine learning community, i.e. the interplay between missing data imputation and the causal structure of the dataset (including the missingness indicators). The paper is well written and motivated. The proposed methodology is clear (in particular Figure 4 is useful to understand the network and optimization process), it seems sound, and it is different to previous approaches. Also, the experimental validation is well structured and is convincing. 

Perhaps what I most miss is a deeper evaluation of the estimated m-graph. The focus of the experimental section is to show that MIRACLE reduces the test RMSE compared to the baseline. Only Figure 7 (left) presents a small discussion on the causal relationships found, although it is still linked to the improvement in terms of RMSE. My suggestion would be to analyse the m-graph estimated in the synthetic experiment, and compare it with the available ground truth. Many interesting questions can be answered there (e.g. are the causal relationships well reconstructed? is the causal regularizer really having an effect on the graph?).

Some other questions and comments that seem relevant:

* MIRACLE is a framework to refine an _existing_ imputation method by learning the causal structure of the dataset. I wonder whether it would be possible to develop (or whether it exists) some algorithm that provides the same functionality but _directly_, i.e. not starting by an existing imputation method that may introduce some bias. That is, I am thinking of an algorithm that simultaneously learns two things: 1) the causal structure of a dataset and 2) how to impute missing values. 

* I was wondering why $[B]_{k,j}$ is defined like that in lines 173-174. I understand that, by doing the partial derivative, you are checking whether the j-th component depends on the k-th variable. And this is indeed what you want for the corresponding item of the adjacency matrix. If that is the case, it may be worth stating it explicitly.

* In the experimental section, there are some incorrect references to the Appendix. For instance, in the last sentence of Figure 5, it says "See Appendix B for MCAR and MNAR results". However, MNAR results are indeed in Figure 5. This wrong references happen several times throughout the experimental section, so please check it.  

UPDATE

After reading the rest of reviews and the authors response, I think that this paper is already in good shape to be accepted. Therefore I keep my score (7). 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper presents a causally-aware imputation algorithm (MIRACLE) which refines imputation of a baseline by simultaneously modeling the missingness generating mechanism. Extensive experiments on synthetic and a variety of publicly available datasets with several baseline comparisons are presented.
main_review: *Very well done paper that leverages the modeling of the missing data mechanism. Along with knowledge of a causal graph, this kind of understanding of the data generating process is strong and should be condoned in data analytic systems.
*The paper is very thorough, figures are clear and elucidate well 
*Very through experiments are presented, systematically varying feature and data size
*It would be helpful to see more discussion on the premise, that causal solutions are expected to correct for spurious correlations introduced by distribution shift due to missing data and preserve the dependencies of the complete data for downstream analysis.  If spurious variables are included (e.g. as parent variables), which can happen in data analyses, then how would those be de-coupled?
limitations_and_societal_impact: Limitations not included.
Authors are suggested to identify limitations of causal graphs and the types of data that can be viewed in such modeling approaches.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors present MIRACLE, a neural network that learns to impute well thanks a specific regularization scheme. The proposed regularisation is motivated by the idea that when imputation is based on all observed data, spurious correlations can arise while it is not the case when it is based solely on the causal parents. Technically, the weights of the imputation network are regularised so that the dependencies between all imputation functions form a DAG. This is possible thanks to prior work which introduced a score based on adjacency matrices that is equal to zero if and only if the corresponding graph is a DAG.  

The neural network proposed by the authors also includes a second regulariser (moment regularizer), this time aimed at matching two different estimators of the first moments of the data. Overall, it minimises a reconstruction loss combined with these two regularisers.

  Extensive experiments are conducted on simulated and real data that compare this neural network to various baselines in terms of imputation quality ( as well as performance on downstream tasks and congeniality in Supplementary materials). On simulated data, it is able to improve imputations over all baselines across all missing data mechanisms. On real data, it does not provide improvements in MCAR, but does so for half of the datasets in MAR and MNAR.

main_review: **Strengths**:
- The idea of using a regularizer that promotes the dependencies among imputation functions to follow a DAG is original.
- Experiments are extensive. In the simulations, the number of samples, features, and missing data mechanisms (MCAR, MAR, MNAR) are varied. Experiments are also carried on 10 real datasets, with varied missing data mechanisms.
- The baselines include recent work (Gain and Optimal-transport based imputations) on top of classical baselines.
- The improvements in terms of imputation performance are clear in simulations and part of the real data.


**Weaknesses and questions**:
- Paragraph 3.3 (Causal Regularizer) is at the heart of the paper. Yet, it is very short and a few things are not clear to me. 
This paragraph describes how the weights of the imputation network can be regularised so that the dependencies defined by f correspond to a DAG).
	- From my understanding, $\partial_k f_j$ is a number and not a vector. Is it the case? If yes the norm used around the partial derivative is a bit misleading.  Also the phrase ‘which can be obtained from the l2-norm of the k-th row…’ is unclear to me. It is a sufficient but non necessary condition for the derivative to be zero? And wouldn’t it be the column instead of the row?
    - Finally, the regularizer 1 (for ‘DAGness’) is about the adjacency matrix B but it is unclear to me what quantity is used for B_{k, j} as a function of the weights of the imputation network. I understand it is a quantity that depends on the weights of the first layer of $f_j$, but I do not understand exactly what quantity and why.
It would be helpful to explain this part a but more.

- It is unclear why MIRACLE works on MCAR simulated data while, as stated by the authors on the real data part, it is not expected to improve performance on MCAR data. On real data, MIRACLE does not improve results compared to baselines in MCAR. The authors explain that this finding agrees the discussion in section 2.1, which states that in MCAR the baselines are not biased. However, the simulations in MCAR in appendix show that MIRACLE brings improvement even if it is MCAR. It seems inconsistent at first sight with the explanation given for the real data. What could be the explanation for this?

- Some important details about the experiments are not given.
    - A downside of using two regularisers is that two hyperparameters need to be set. How are they set in the experiments?
    - What sample sizes are used for the train and test sets?
    - In the MAR and MNAR simulations, I understand that 30% of features have 30% missingness, is it correct? That would lead to a much smaller amount of missing values than in MCAR, in which I understand all features have 30% missing values.

-  I think it is important to understand the relative importance of the two regularizers, which are very different in nature. It is nice to have the ablation study for the real data, however I think it would give more insights into the relative importance of the regularisers on the simulated data. Notably, should we expect that the moment regularizer, which is derived for the MAR case and not appropriate for the MNAR case, help in the MNAR case?

limitations_and_societal_impact: -
review_title: null

Meta-review: The reviewers have provided thoughtful and constructive comments. They have responded to the authors' feedback and the most active reviewers who are championing this manuscript continue to lean towards acceptance. I hope the authors will take the reviewers' comments to heart and encourage them to incorporate their thoughts in preparing the camera-ready version of their manuscript.
Decision: Accept (Poster)