Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors mainly study the problem of missing data in treatment effect estimation and highlight the importance of addressing this problem. The authors propose a selective imputation scheme which is more well suited for addressing missingness in such scenarios. Authors also present several sample scenarios illustratively which indicate potential issues with current methods while doing TE with missingness.  Empirical results compare different scenarios where general imputation schemes (imputing all data, no data, wrong data) can be much worse than their proposed method.
main_review: **Strengths of paper**

- Paper presents a very elaborate writeup with a detailed introduction to causality and missingness in treatment effect estimation which makes it a good read for audience who are new to this field.

- Problem addressed is a relevant problem and is applicable across several real-world problems.

**Weakness**

- The paper has a lot of content on introducing causality which instead could have been used to add more use cases on how selective imputation can be useful for TE
- It does appear that the authors could have studied the exhaustiveness claim of MCM in depth to tease out the contributions of this paper better. This is a very broad claim with insufficient justification in the current draft.
- It is not exactly clear how this paper advances the technical state-of-the-art to clear the ICLR bar of acceptance. The general writeup and contributions make it a better fit for a venue like UAI and other similar avenues.

Overall, I would like the authors to think more broadly on the imputation problem itself as industry problems rarely rely on using any kind of imputation method (mainly due to skepticism and avoiding corrupting the data). It might help if authors can talk about broader adoption for this work for such a real-world setting.
summary_of_the_review: I have highlighted my major observations from this paper above. While the paper is written in a very elaborate manner, I do believe it is still not clear if the technical contributions are advancing the state-of-the-art to clear the ICLR bar of acceptance. Authors should tease out the technical contributions around exhaustiveness and others more clearly and remove big sections on assumptions, metrics (can be written briefly with citations). I would also recommend trimming down the graphs to only keep the most relevant DAGs with a clear messaging.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies dealing with missing values in estimating treatment effects. Authors identify a new missingness mechanism, mixed confounded missingness (MCM), including missingness that determines treatment selection and missingness that is determined by treatment selection. The authors show that both imputation and no imputation lead to poor treatment effect estimations. The authors present a selective imputation strategy that informs which variables should be imputed and which should not. They empirically demonstrate the effectiveness of the strategy.

main_review: Authors use graphic causal models to model the various missingness in causal effect estimation. This is an interesting exercise.

However, I doubt the correctness of mapping Figure 1(d) to Figure 3. In Figure 1(d) the relationships between the rightmost four variables are "W \to Z_2 \to X'_2 \leftarrow X" (Note that I use Z_2 and X'_2 to replace the superscripted Z and X in Figure 1(d)). We should understand the relationships in the following way. W determines the missing values of Z_2 and subsequently the observed values of X'_2. This does not mean that W will determine X_2 when it does not include missing values. Using the authors' example, the participants of the job training program provide additional information whereas the non-participants do not. So, the missing values in X'_2 are determined by W. However, if we have a magical way to collect the additional information from non-participants (i.e. the missing values are imputed perfectly). The edge W \to real X_2 will be broken. Figure 1(d) becomes Figure 1(c). If X_2 is imputed correctly, there is not an edge W \to X_2 and the discission before "selective imputation" on Page 7 is invalid.   

Given that Figure 3 is confusing and leads to an incorrect conclusion. There is a soundness problem with the paper.

Since the experiments follow Figure 3, the results cannot be trusted. If X_2 is a part of the covariate set, there is no edge W \to X_2 without missing values. The causal effect on the set of X is unbiased. If X_2 is imputed correctly, the causal effect is also unbiased. 

Interestingly, there is not a discussion of how the missing values are imputed.
    
summary_of_the_review: There is a soundness problem with the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: "we identify a new missingness mechanism, which we term mixed confounded missingness (MCM), where some missingness determines treatment selection and other missingness is determined by treatment selection."

The author gave a new term called “MCM” which is not something new. MCM is a type of MNAR (Missing-Not-At-Random). 
MCM cannot be listed parallel to MCAR, MAR and MNAR. 


main_review: The questions they are trying to solve is an important problem.
For example in the discussion the author emphasize that "more care and thought should
be put into imputing missing data when estimating treatment effects." which I agree with completely.

The authors are using simplified DAG graphic tool to modeling the exposure, mediator, outcome variables with missingness and propose the causal relationship of missingness. In the real world data, one covariate may cause missingness of an exposure, which could be the predictor of the outcome variable. However, the missingness of this covariate may have no direct correlation with outcome variable. Therefore, to determine which variable will be imputed or not will not be simply determined by strategy will be limited.

Missingness in one variable may have direct or indirect relationship with missingness in the other variable or outcome variable. Alternative speaking, this correlation may not be linear. Which variables will be imputed or not imputed for the prediction of outcome variable will be  determined using machine learning approach for the feature selection.

Finally, even though it is difficult to use real word data for experimentation, it is important to attempt to do that. The imputation procedure has to be designed for a specific field as pattern of missingness may be very different among disciplines. The pattern of missingness and mechanism for a given dataset should be recognized. Whether the competition is favoring a certain method or procedure has to be determined in the “real-world” data with “real-world” missingness by considering recognized and unrecognized missing pattern/mechanism, as well as the plausible distribution of missing data. Testing a method on synthetic data, with no regards to observed patterns of missingness may only add noise to the field. A recent paper in the field was just published this month: https://www.nature.com/articles/s41746-021-00518-0 

Therefore, even though the idea of improving imputation is excellent, the logic used in this study is not robust.  
summary_of_the_review: Even though the idea of improving imputation is excellent, the logic used in this study is not robust and does not take into account true realistic situations.  

Use of only synthetic data, when we have access to many real-world datasets is not acceptable. How the synthetic data was created has a direct effect on how the methods will work. It is possible to use real-world data and introduce additional hold-out values to evaluate the methods. see for example: https://www.nature.com/articles/s41746-021-00518-0

The importance of real-world data is key here because the whole idea behind this paper is that missingness patterns are not completely understood.
review_title: null

Meta-review: In this paper, the authors propose a new type of (missing not at random) model they call the MCM (mixed confounded missingness).
The authors further discuss that given their model, naive imputation strategies do not work, and a model-tailored imputation strategy is needed.

The reviewers did not receive the paper favorably, with main complaints centering around: (a) outlining novelty compared to existing approaches to missing data, (b) whether imputation is a good strategy for dealing with missing data, and (c) whether the paper's results are actually sound.

Here's my perspective on these worries.

The paper aims to deal with missing data in a causal inference context (in other words, the target of inference is a causal effect, and our data happens to have entries missing not at random).  Further, the paper aims to work within a graphical modeling formalism for missing data models.  Finally, the paper points out that imputation is to be done with care if data is missing not at random (a point both myself, and reviewers agreed with).

Areas of improvement in the paper, in my mind, would be: (i) better literature review and putting authors' work in context of prior work, (ii) being clear about identification, and (iii) discussion of estimation strategies (not just imputation).

Dealing with missing data (in particular right censoring, but also more general types of missingness) in causal inference is a very old problem, with an established literature in statistics and public health.  In fact, methods for dealing with both causal inference and missing data together are a part of standard graduate curriculum in epidemiology and biostatistics in many Universities.

(i) Literature review and context.  Some papers the authors may find helpful to review:

James M. Robins, Andrea Rotnitzky, Daniel O. Scharfstein.  Sensitivity Analysis for Selection bias and unmeasured Confounding in missing Data and Causal inference models.  Part of the The IMA Volumes in Mathematics and its Applications book series (IMA, volume 116).

This paper discusses lots of relevant things, but in particular sensitivity analysis methods to violations of MAR in settings the authors worry about.

James M. Robins. Non-response models for the analysis of non-monotone non-ignorable missing data. Statistics in Medicine, 16:21–37, 1997.

This paper is an early example of an MNAR model that may be represented by a directed acyclic graph.

Karthika Mohan, Judea Pearl, and Jin Tian. Graphical models for inference with missing data. In C.J.C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 1277–1285. Curran Associates, Inc., 2013.

Ilya Shpitser, Karthika Mohan, Judea Pearl.  Missing data as a causal and probabilistic problem.  In Proceedings of the Thirty First Conference on Uncertainty in Artificial Intelligence (UAI-15), pp. 802-811, AUAI Press, 2015.

Rohit Bhattacharya, Razieh Nabi and Ilya Shpitser. “Full Law Identification In Graphical Models Of Missing Data: Completeness Results.” In Proceedings of the Thirty-Seventh International Conference on Machine Learning (ICML-20), pp. 7153-7163, 2020.

These papers deal with general models of missing data using graphs.

Since the authors use graphical models as well, I urge them to put their contribution in context with this prior work.

(ii) Identification.  The authors should clearly discuss whether treatment effects are identified under their model, and if so, by what function.  If this function is not closed form (which can happen in missing data), this should be discussed as well.  This should be contrasted with other missing data work that derives identification under MNAR, particularly using graphs.

(iii) Estimation.  The authors chose to use imputation.  Imputation is a sampling approach to inference in missing data.  Others include maximum likelihood or Bayesian methods (via EM), or semi-parametric inference via influence functions.  If the authors chose to concentrate on imputation, specifically, they should explain why (as other methods have noted advantaged, e.g. statistical efficiency, quantification of uncertainty, etc.).

Cautioning against naive imputation is a fine thing to do, but everyone working on missing data problems already knows naive imputation does not work for MNAR data.  Please do not oversell your contributions.  Saying things like: "MCM being the first formalisation of a missingness mechanism when there are treatments at play." is neither true, nor helpful for the peer review process.

With all that said, the MCM model has the potential to be an interesting MNAR model, and placed in proper context of existing work, could be a very interesting addition to the missing data literature.  However, the draft needs a bit more work before it is ready for publication.
Decision: Reject