Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a method to more optimally learn deep learning classifiers in the presence of missing values. More specifically, the authors present the approach called supervised missing data importance-weighted autoencoder (supMIWAE) bound which allows them to train a joint model of covariates and outcomes by marginalizing over the missing values. 
main_review: The paper is written clearly and there are no obvious errors I can see in the theory or the experiments presented in this paper. However, I have not checked some of the derivations in detail. The method presented in the paper is largely based on previous work on variational inference, but it is otherwise novel and noteworthy theoretical contribution. Their application to train a joint model of covariates and outcomes with DLVMs and neural classifier in the presence of missing data is a fairly novel approach to my knowledge. The claims made in the paper are well backed by experiments and therefore the results are quite solid. In terms of impact, the problem of missing values is a very ubiquitous one and therefore the potential impact of the paper is significant. Moreover, the results presented by the authors indicate that their method outperforms other relevant approaches (single imputations methods) in a restricted number of scenarios: low-capacity regimes, or when the discriminative model has a strong inductive bias.  
summary_of_the_review: Noteworthy application of variational auto-encoders to improve classification in the presence of missing data. Overall a solid paper, I recommend accepting the paper in the conference.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose supMIWAE, a supervised missing data importance-weighted autoencoder, to address the challenges of training with missing data. They claim that the new approach, a VAE combined with a discriminator, jointly trained, is more scalable, and performs better than existing data imputation methods, in particular MIWAE (Mattei & Frellsen, 2019) 
main_review: Strengths:
+	The proposed method offers end to end training, which is useful for scalability and transferability. They claim that such a method can be applied to discriminators. 
+	authors have both visually and empirically shown improvements in data imputation for MNIST and Fashion-MNIST
+	It is shown to perform better than MIWAE, (Mattei & Frellsen, 2019), the main baseline for this works

Weaknesses:
-	While there are some gains in performance, it feels incremental both in % increase and scale. MNIST and FMNIST are very similar small, almost binary valued datasets, and weakly supports the claim of scalability.
-	In UCI datasets, where features are real numbers, and probably contain more information per feature, supMIWAE performs at par with MIWAE.
-	Most aspects of the technique are leveraged from the prior works. 

General comments:
●	How does the supervision and joint training impact the training complexity of the system?
●	Interestingly, supMIWAE performs definitely better than MIWAE for MNIST, and FMNIST, however such gains are no longer visible for UCI datasets. The gains of joint training seem to be limited to a certain type of data. 
●	In terms of practical applications, where do the authors see their method bring the most gains? Image, language, or statistical datasets. 


summary_of_the_review: The addition of supervision and joint training to MIWAE seems novel, and while the work improves upon the prior methods of using DLVM for missing data, it falls short in the claim of scalability and overall gains in performance across different types of datasets.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper handles the issue of missing values in supervised deep learning settings. The fig.1 describes their method aptly. Their method (supMIWAE) is a combination of a VAE with a neural network classifier. Given the task to predict p(Y|x_{obs}, x_{miss}), the authors view it as a joint model of covariates and outcomes. Their model takes in x_{obs} and first fits a distribution to get x_{miss} and then models them as a joint distribution to predict outcome Y|(x_{obs}, x_{miss}). They use importance sampling technique to get multiple samples from the generative model. 
main_review: Pros:
1. The authors do a good job of explaining the challenges of training discriminative models with missing data as well as going over the related works.
2. Their approach is modular and extendable to many different choices of neural network architectures. 
3. The paper is well written and easy to follow. Good work there!

Concerns and questions:
1. Are eq. 13, 14 novel contributions of this work? Or these were previously derived and new to the proposed missing value handling settings?
2. The novelty of the work is marginal. The multiple imputation idea, viewing the imputation problem as joint distributions (eq. 1,2) and also going through a few of the related works mentioned in this paper, I feel that most of the ideas proposed in this work are already out there. This work finds a combination of them which works well but the experiments are not extensive. 
3. The experiments are restricted to very small datasets. Would definitely like to see how this method performs on scale. As, I can foresee some issues with the training as it uses sampling. 

summary_of_the_review: The work is good but not novel enough. Need to show results on larger datasets. Please also highlight the fail cases in the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper approaches the problem of supervised learning with missing data. The authors propose a probabilistic approach by jointly modeling the observed data, missing data and outcomes. The main contribution of this model is that they rely on deep generative models which seems to be an improvement from previous methods based on simpler generative models like mixture of Gaussians, for example. With the proposed model, the authors derive an optimization problem that consists in optimizing the discriminative model (classifier) and the generative model simultaneously. For the training and testing phases, the proposed algorithm considers multiple imputations of missing data, which is demonstrated to be superior to single imputation methods. Experimental results on small and simple datasets (2D datasets, MNIST digits-Fashion and regression) are nicely presented and analyzed.
main_review: Strenghts:
- Strong theoretical foundations
- Clearly written with nice illustrations of the used principles.
- Good review of previous approaches to the supervise learning with missing data problem

Weakness:
- While the authors claim that their method is for deep learning, the provided examples are given on small datasets and used neural networks are not very deep. For example, a 4-layer CNN is used for MNIST datasets. It is not clear if the proposed method could be used on bigger datasets and deeper architectures.
- A comparison of computational time with respect to other methods is missing, which makes to think that maybe this method is too expensive to implement for big datasets and deep learning architectures.
- Experiments do not contain comparison with previous methods other than imputation based ones. There are recently published methods cited by the authors that provided implementation code for learning classifiers and missing data at the same time, i.e. avoiding the imputation followed by classification approach.

summary_of_the_review: A good paper including very clear and technically sound principles for solving the problem of supervised learning with missing data. Experiments are illustrative although it would be needed to implement them on bigger datasets and deeper neural network architectures as well as to compare to other methods in the literature.
review_title: null

Meta-review: This paper addresses features that are missing at random in deep supervised learning, especially regression and classification. A deep latent generative model is trained in conjunction with a discriminative model, so that the distribution of the covariates is properly modeled and allows efficient variational inference for imputation.  Superior performance is achieved in low-capacity domain or when strong inductive bias is present in the discriminative model.

The paper is well written, with solid empirical support.  The approach is also generic.  Although there is some concern on novelty, overall this paper appears a solid contribution and is a good addition to the proceedings.
Decision: Accept (Poster)