Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper is about identifiability of the data generating process and the missing value mechanism in a missing-not-at-random setting. The paper provides an analysis of identifiability of generative models and propose a deep generative model with identifilibity guarantees under some assumptions. There are existing methods for handling MNAR without identifiability guarantees and there are theoretical analyses on identifiability without practical scalable algorithms. The paper aims to bridge this gap.  
A VAE based model for imputation is proposed and compared to two other baselines.  The proposed method models the data and missing mechanism jointly, and in order to ensure identifiability the proposed method needs auxiliary variables that are always observed.
main_review: The paper is well written and clearly presented. The focus on MNAR is important as it has been much overlooked compared to how prevalent it is in real world data and the focus on identifiability is useful to further the use of models handling MNAR data.  
It nicely address the challenges of MNAR and identifiability and builds up its case coherently. Identifiability is first analyzed under the correct graphical model, then relaxed to cover data model mismatch. The model assumptions are clearly stated as well as the assumptions about the data generating process.

Identifiable models makes sure that the model parameters can be uniquely determined, that the parameter estimates are unbiased.
- In line 38 and 120 this is taken to also be the case for imputations from such models, that identifiability is needed to achieve unbiased imputations. Is this well established? could you provide any references on this? Of course if you indeed get unbiased estimates of the ground truth parameters, imputations will be unbiased, but couldn't two (non-indentifiable) models with different parameter settings in principle give the same (unbiased) imputations? Or is it that non-identifiability CAN lead to biased imputations? 

The experiments are few, but nice. Figure 3 is informative as a model evalution using another metric than imputation MSE. 
- Is it clear that the difference between GINA and not-MIWAE is due to identifiability and not just plain old model misspecification? that is, the missing mechanism is based on the latent variable, GINA uses the latent variable as input to the missing model while the not-MIWAE does not. Could the problem of model misspecification be separated from the issue of non-identifiability in some way in this experiment?

Minor comments:  
- Eq (1) is referred to as the selection model factorization, but this is just the joint (which could be factorized as selection model or pattern mixture), maybe update text or equation.   
- Line 177: should it be proposition 1 instead of proposition 2?  
- Maybe this work [1] on MNAR is also of interest to you.  
- Appendix B), I think you mean the the not-MIWAE is described by figure 1 d), not figure 1 a) (and the PVAE is described by figure 1 a)). 

[1] Ghalebikesabi, Sahra, et al. "Deep Generative Missingness Pattern-Set Mixture Models." International Conference on Artificial Intelligence and Statistics. PMLR, 2021.
limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The article is concerned with model identifiability in Missing Not At Random (MNAR) data setting, with a particular focus on deep generative models. The authors introduce a unified view of model identifiability under MNAR, and derive sufficient conditions for identifiability in several MNAR scenarios, which highlight new important concepts. They also introduce a new imputation method for MNAR data based on identifiable VAE models, which is submitted as a python code, and extensively tested on synthetic and real data sets against state of the art methods for MNAR imputation. 
main_review: Identifiability of MNAR data models is an important topic, as MNAR data is a major problem in applied problem which often impairs analysis, and few methods have been developed to overcome this. Even fewer existing work provide some theoretical understanding in such settings. In addition, existing results on identifiability usually lead to computationally costly imputation methods which are not easily generalized. Thus, I think this article is important and quite remarkable, as it provides original theoretical results, along with an efficient software and exhaustive numerical simulations, as well as an actual application to a large scale education data set; I believe the method could have an important impact in applications. I have a few questions for the authors
- Concerning Assumptions 2 and 3, is there a dependence somewhere in your results in the number of partitions S and L and in their size ? 
- Do you believe these identifiability results could pave the way for estimation consistency in MNAR settings ?
- What kind of architecture did you use in practice for the VAE in GINA ? Did you use different ones for each problem ? How robust do you think the procedure is to the choice of architecture ?

I have read the authors' rebuttal, which provide relevant answers to my questions. The authors added simulations to empirically evaluate the robustness of the method to the networks' architecture.
limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a new approach to identify the generative model under missing not at random scenario.
The paper starts with a model on the underlying distribution and then propose assumptions so that the model's parameters are still identifiable even if missing at random is violated.
main_review: This paper proposed an interesting theoretical framework on the missing data problem. While I see some novelty in this paper, there are some critical issues that the authors did not address.

Q1. [missing data vs model assumption]
In missing data problems, there are two types of assumptions we need to consider: missing data assumption and model assumption. 
This paper starts with a (parametric) model and then impose missing data assumption on it.

While this leads to an identifiable result (under additional assumptions), this identification is not "nonparametric identification". 
I think this should be discussed in the paper.


Q2. [Strong assumption (A1-3)]
It seems to me that the main idea that this paper can avoid problems of MNAR is due to assumption (A1-3), which basically imply the problem is similar to the ignorable scenario (MAR). 
These assumptions seem to be pretty strong.

Can the authors provide some concrete and simple examples on how these assumptions can be true? 
Will the data follows a multivariate Gaussian with some additional constraints make these assumption true?


Q3. [Section 3.2]
This relaxation is rather weak--the one-to-one assumption almost says that the problem remains the same. 
What generally would happen is that the data is not generated from a parametric model but a nonparametric model.
In this case, the model we assume will be a best parametric approximation to the target model. 
Can we argue that even under this case, our fitted model will be close to the nonparametric model with some bounds on the difference? 


Q4. [About GINA]
I was not very comfortable about this part since the neural network itself is a model and is a very complex one. 
What are the conditions on f? This should bee clarified. 

Also, in appendix G, the authors basically need the generative model to be from the model assumed by GINA.
But it is pretty difficult to believe that a neural net is the actual generative model of the data (neural net can be a good approximation but not the actual generative process)...

Moreover, I don't think that the authors proved GINA satisfy assumption A4 in appendix G. 
There needs to be some additional assumptions on the actual generative process when the neural net is not the actual generative model. 

Q5. [Model diagnostic]
Given that the authors have assumed a particular model for the generative process, the first thing that we can do is to check if the model assumption makes sense (at least for the observed part).
This should be discussed.
And what should we do if we find out that our assume model does not fit to the data?

limitations_and_societal_impact: I have described limitations of the paper in my main review.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies identifiability of generative models for Missing Not at Random (MNAR) data. MNAR is the most general missing mechanism, where the the missingness can depend on both observed and unobserved data. Most prior imputation techniques work under MCAR or MAR assumptions, and there are few models to handle missing data under MNAR. In this respect, model identifiably under MNAR assumption has been mostly overlooked. In fact, the identifiability of models is crucial for learning from MNAR data to uniquely determine the model parameters and have unbiased imputations. This paper provides the sufficient conditions and assumptions to satisfy identifiability of generative models under MNAR. In addition, it proposes a practical deep generative model (GINA) based on VAEs which can satisfy the identifiability conditions.

The contributions are as follows:
* Theoretical analysis of model identifiably under MNAR, which gives sufficient conditions to have identifiable models (uniquely identify the model parameter, matching the ground-truth data generation process).

* A practical algorithm to satisfy these conditions and learn deep generative models under MNAR.

* Experimental studies showing effectiveness of the proposed model on both synthetic and real datasets.
main_review: ### Strengths
- I appreciate the author’s great effort to include both theoretical analysis and the practical model.

- Most imputation papers try to blindly propose a model which gives better results. This paper originates from an important limitation in existing imputation techniques and tries to fill the gap in the state-of-the-art models. Imputation under MNAR and specially model identifiability under MNAR to have unbiased imputations have been mostly overlooked in the prior work. So, I believe this work has significant contribution to the literature for learning from incomplete data.


### Limitations
- I appreciate the experimental studies by the authors on new datasets which are more relevant to MNAR problem. I think the experiments are sufficient. But, as a recommendation for future work, the UCI datasets in [10] or the paper below (e.g. Table 4) could be also used to have a more comprehensive comparison.
    - Yu Gong, et al. "Variational Selective Autoencoder: Learning from Partially-Observed Heterogeneous Data." International Conference on Artificial Intelligence and Statistics, 2021.
    - BTW, this paper can also work under MNAR. But, it does not analyze the model identifiably. It is good to cite this paper and explain the advantage of your work.

### Minor comments
- The paper is somehow hard to read. 
    - Sometimes the notation is not conventional and makes it complicated to understand. For example, using $\mathbf{x}$ for random variables is more conventional compared to $X$. I understand that the authors used $X$, $R$, $O$ as sets. But, defining them as sets makes it hard to understand.
    - GINA algorithm is hard to understand. It is not based on the original VAE formulation. So, it is hard to understand and requires reading [14] and [10].
    - The main limitation of the model is discussed in two sentences in Section 7. It was hard for me to understand it. What does the auxiliary variables mean?
    - There are also some typos:
         - The sentence in line 104-105 is not understandable. 
         - In line 158, I think $R_{I\O'_l} = 0$ miss $= 0$


### After Rebuttal
I acknowledged I read the authors' response and considered it in the final discussions with the reviewers.

limitations_and_societal_impact: There is no sensitive data in this paper. But, in general, imputation techniques might be used to retrieve the users' information without their consent.
review_title: null

Meta-review: 
The paper is on imputing missing variables when they are missing not at random (MNAR). All reviewers agree that the paper makes a valuable contribution on a topic that is important but often overlooked. This is a clear accept. 

In the camera-ready version, please incorporate the reviewers' feedback. Moreover, during discussion, reviewers brought up the idea of illustrating assumptions A1-A3 on a simple example, which I would indeed recommend to do (e.g in the appendix).

Decision: Accept (Poster)