Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper studies a very interesting phenomena in machine learning called VSP, that is the output of the model is highly affected via the level of missing values in its input.  The authors demonstrate the existence of such phenomena empirically, analyze the root cause for it theoretically, and propose a simple yet effective normalization method to tackle the problem. Several experiments demonstrate the effectiveness of this method.

In general I think the paper is descent and elegant. It is motivated from real-world pain-point, gives a rigorous study towards the root cause, and the proposed method is very effective. To the best of my knowledge there is no prior work looking deep into this area and this paper does bring new insights to the community. As a result I would vote for its acceptance.

One issue is that I find the backbone methods in experiments are somehow out-of-date. For example, AutoRec (2015) and CF-NADE (2016). I admit that I’m not an expert in the field of recommendation but still think that more recent, and powerful baseline algorithms should be applied on to further demonstrate the true effectiveness of Sparsity Normalization.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper provides a novel solution to the variable sparsity problem, where the output of neural networks biased with respect to the number of missing inputs. The authors proposed a sparsity normalization algorithm to process the input vectors to encounter the bias. In experiments, the authors evaluated the proposed sparsity normalization model on multiple datasets: collaborative filtering datasets, electric medical records datasets, single-cell RNA sequence datasets and UCI datasets. Results show that the proposed normalization method improves the prediction performance and the predicted values of the neural network is more uniformly distributed according to the number of missing entries.

The paper describes a clear and specific machine learning problem. Then the authors demonstrate a simple normalization strategy is capable of fixing the issue of biased prediction. The paper has a well-organized structure to convey the motivation. Therefore, my opinion on this paper leans to an acceptation. My questions are mainly on the experiment section:

1) As shown in Table 2, there are various new collaborative filtering methods proposed after 2015, why the authors chose to extend AutoRec (Sedhain et al., 2015) but not other new methods?

2) In the experiments, you compare your model with zero imputation (Please correct me if w/o SN is not zero imputation). However, I think it is a common practice in machine learning that we perform imputation with mean or median values. I'm interested in knowing whether filling with mean/median values work with these datasets.

3) In section 4.5, you mentioned that "SN is effective
even when MCAR assumption is not established". However, I'm still not clear about the reason. I believe many machine learning datasets have NMAR (not missing at random) type of missing data, but not MCAR. So this is an important issue for me.

4) Does your model assume all input values are numerical but not categorical? 
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Zero imputation is studied from a different view by investigating its impact on prediction variability and a normalization scheme is proposed to alleviate this variation. This normalization scales the input neural network so that the output would not be affected much. While such simple yet helpful algorithms are plausible there are number of remaining issues:
1-	Zero imputation, as authors mentioned, is not an acceptable algorithm for imputation and improving on that via the normalization proposed in the paper cannot be counted as an exciting move in this area unless an extensive comparison shows it’s benefits over the many other existing techniques. I am interested to see how would the results be if you compare this simple algorithm with more complicated ones like GAIN or MisGAN. It is argued in the paper that with high dimensional data, your algorithm is more acceptable, but how would it be with in other cases?
2-	Your algorithm is only explained with neural net framework, how can we extend it to the other machine learning models?
3-	Is batch normalization used in your experiments? Scaling the activation in one layer to reduce its impact on the next layer is somehow similar to what happens in batch normalization, and I am wondering if BN makes any similar effect?
4-	Please provide labels for the x-axes in the figures.

------------------------------------------
After rebuttal:
Thanks for adding the extra experiments.
Looking at Table 9 in appendix, I am bit surprised to see that sometimes mean imputation works better than MICE (GAIN usually works good with large data). Maybe it attributes to the missing features. How did you choose to apply 20% missingness? randomly?
review_title: Official Blind Review #1

Meta-review: This paper investigates the problem of using zero imputation when input features are missing. The authors study this problem, propose a solution, and evaluate on several benchmark datasets. The reviewers were generally positive about the paper, but had some questions and concerns about the experimental results. The authors addressed these concerns in the rebuttal. The reviewers are generally satisfied and believe that the paper should be accepted.
Decision: Accept (Poster)