Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a method to impute missing features using a generative model and train a predictive model on top of imputed dataset to improve classification results. They first train a GAN model where the generator outputs an imputed representation of the input and discriminator is trained to predict if an individual features (such as a pixel) is imputed or not. Given the generator and incomplete sample, they train a predictor using the output of the generator, imputed sample, as input. Their main contribution is using a MC averaging to compute the prediction by repetitively sampling from the noise variable, z, and generating different imputations from generator. They show that the proposed model improves upon the previous SOTA on final classification performance.

Overall the paper is clearly written. But I do feel it is a bit incremental over the GAIN approach. The overall GAN architecture is very similar to GAIN's and although stochastic prediction shows clear improvements it is a bit straightforward. However, I think the uncertainty of the imputations and its effect on the final prediction is interesting. I suggest the authors to extend this part with more detailed analysis.

There are several parts that are confusing/missing in the paper:

- In GAIN, they use a hint vector as an input to the discriminator. They show that without the hint vector, there is no unique solution (this is shown without the MSE loss). The authors do not use this vector in their approach (as in Figure 1) and it is not clear to me if it causes any instabilities or if multiple experiments yield similar results or if the stochastic prediction benefits from this.
- On what type of examples GI is more accurate than other models? Since stochastic prediction is the main difference from GAIN, is this related to the multi-modality of the noisy examples?
- Can you explain the difference between the results in Figure-7 and Table-2? Results between the two mismatch.
- I think the statement in the first paragraph in Section 4.4 that "MSE loss term would act as a denoising loss smoothing noisy missing pixels" could be misleading. MSE is used with mask in GAIN, hence it only applies to the observed features during training. Its effect on smoothing noisy missing pixels is not clear.


I think the paper would benefit if the authors could explain/show:
- Increasing the missing rate would also increase the possibility that the ground truth be a more multi-modal distribution. Especially in rectangular generation part where it can remove a complete object. Does stochastic averaging benefit more in this case?
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This is a nice piece of incremental work on top of previously published GAN imputation methods. It seems to work well in the limited evaluation and is at least claimed to be easier to use for practitioners. This paper could benefit tremendously from both better evaluation and discussion. The paper would be much clearer if GI contextualized itself relative to GAIN on the one hand (which is the most similar GAN method) and multiple imputation on the other hand (of which this is almost, but not quite, an instance of). 


Suggestions for improving the introduction & discussion: 
* The purpose of this paper is to model uncertainties about missing values — you really should say more about probabilistic methods than "A few exceptions exist such as Bayesian models”.  At least give some motivation for why certain imputations problems couldn’t be feasibly solved by modeling the missing values in a probabilistic programming framework. 
* Other GAN methods for imputation (GAIN and MisGAN) are dismissed as "often very complicated to be applied in practical setups by practitioners”. Given that the described method resembles GAIN, is it really much simpler? If so, can you be more specific when characterizing related work?
* "This is different from approaches such as multiple imputation where several predictors are trained on different imputed versions of a dataset.” — the main difference between this approach and MI is that you’re interleaving imputation and training a downstream model. Emphasize this earlier on, since it will make the whole technique easier to understand.

Suggestions for improving the evaluation:
* You’re imputing missing rectangles from an image dataset — please show us the resulting images. I would greatly prefer this to Section 4.5 — which is a very low sample size, low dimensionality example and it’s really unclear how well it generalizes to real data. 
* "We also considered using root means squared error (RMSE); however, we decided not to use this measure as we observed an inconsistent behavior using RMSE in our comparisons as RMSE favors methods that show less variance rather than realistic and sharp samples from the distribution.” — I think this is a mistake, likely motivated by the proposed method doing worse under the RMSE metric. Show us several relevant metrics and then discuss their tradeoffs afterward. 
* "We run each experiment multiple times (at least 4)” — please report how often each experiment was run, even better if you standardize this number. 
* For Table 2, please provide accuracy without missing values as a baseline.
* Add MICE or some other “standard” imputation method as a baseline. 

Suggestions for improving readability: 
* Many sentences start with “in this” (e.g. “in this case”, “in this setting”, &c). Sometimes these sentences even co-occur within the same paragraph. Try to switch up the phrasing and move away from repetition. 
* Not a complete sentence: "For instance, jointly training multiple generator/discriminator networks, tuning objective functions with multiple hyper-parameters, etc."

Update: I think the latest draft of the paper is a big improvement, the inclusion of a "classical" baseline, improved language and additional appendices are all welcome. I'm leaving the rating as a "weak accept" since the paper still feels rough and could use additional editing/streamlining. 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The uncertainty of having a missing value is investigated on the prediction by not assigning a single imputed value but N different values generated via an imputer network (based on GAIN). Unlike old-fashion multiple imputation techniques, one predictor is trained on different samples and it induces the uncertainty. The experiments on number of datasets show the proposed predictor is capable of having a fairly better performance.
Overall, this paper raises an interesting point about missing data imputation via generative models, and well-written; however, there are number of concerns:
1-	The predictor is trained on different version of imputed samples (imputed via the generator); this equates to making noisy version of the real samples, where noise is applied to the missing variables. A side effect of this is generalization of the predictor; thus, have you been careful that the improved accuracy is not due to generalization? In other words, if we imposed the generalization via adding gaussian noise to the imputed samples by GAIN for example, would we get improved accuracy too?
2-	Your method known as GI is a modified version of GAIN. You could also use MisGAN, and I am wondering if the results would have been different if generator in MisGAN was used in GI in Figure 2 (b), as MisGAN works better than GAIN.
3-	I am also wondering how the GAIN imputation changes by removing the MSE term? GI discards the MSE term in GAIN, and it changes the distribution of the imputed variables by GAIN. Could you maybe fit a Normal distribution on a chosen imputed variable  (N=128) and visualize how different it is from the distribution of imputed variables with GAIN with MSE term. 
4-	In justification for claim 1, it is said “This is equivalent to training models using noisy labels”. This is not accurate: in noisy label prediction, we have one (noisy) y corresponding to each x, in your case there are multiple ys for one sample.
5-	In the implementation details, I cannot fully wrap my head around the part “z vector of size 1/8”; how did you choose this 1/8?

review_title: Official Blind Review #2

Meta-review: The paper proposes a method that does uncertainty modeling over missing data imputation using a framework based on generative adversarial network. While the method shows some empirical improvements over the baselines, reviewers have found the work incremental in terms of technical novelty over the existing GAIN approach which renders it slightly below the acceptance threshold for the main conference, particularly in case of space constraints in the program. 
Decision: Reject