Why Not to Use Zero Imputation? Correcting Sparsity Bias in Training Neural Networks
Authors: Joonyoung Yi
Juhyuk Lee
Kwang Joon Kim
Sung Ju Hwang
Eunho Yang
Venue:  (2020)
Keywords: Missing Data
Collaborative Filtering
Health Care
Tabular Data
High Dimensional Data
Deep Learning
Neural Networks

Abstract: Handling missing data is one of the most fundamental problems in machine learning. Among many approaches, the simplest and most intuitive way is zero imputation, which treats the value of a missing entry simply as zero. However, many studies have experimentally confirmed that zero imputation results in suboptimal performances in training neural networks. Yet, none of the existing work has explained what brings such performance degradations. In this paper, we introduce the variable sparsity problem (VSP), which describes a phenomenon where the output of a predictive model largely varies with respect to the rate of missingness in the given input, and show that it adversarially affects the model performance. We first theoretically analyze this phenomenon and propose a simple yet effective technique to handle missingness, which we refer to as Sparsity Normalization (SN), that directly targets and resolves the VSP. We further experimentally validate SN on diverse benchmark datasets, to show that debiasing the effect of input-level sparsity improves the performance and stabilizes the training of neural networks.

Published as a conference paper at ICLR 2020
WHY NOT TO USE ZERO IMPUTATION? CORRECTING SPARSITY BIAS IN TRAINING NEURAL NETWORKS
V Juhyuk Lee', Kwang Joon Kim², Sung Ju Hwang 1.2 Eunho Yang 1.2 KAIST AITRICS", Yonsei University College of Medicine", South Korea (joonyoung.o yii, sehkmg, sjhwang82, eunhoy) @ka ikaist.... ac kr, preppiełyuhs.ac
Joonyoung Yi',
ABSTRACT
Handling missing data is one of the most fundamental problems in machine learning- Among many approaches, the simplest and most intuitive way is zero imputation, which treats the value of missing entry simply as zero. However, many studies have experimentally confirmed that zero imputation results in suboptimal perfor- mances in training neural networks. Yet, none of the existing work has explained what brings such performance degradations. In this paper, we introduce the vari- able sparsity problem (VSP), which describes phenomenon where the output of a predictive model largely varies with respect to the rate of missingness in the given input, and show that adversarially affects the model performance. We first theoretically analyze this phenomenon and propose a simple yet effective technique to handle missingness, which we refer to as Sparsity Normalization (SN), that directly targets and resolves the VSP. We further experimentally validate SN on diverse benchmark datasets, show that debiasing the effect of input-level sparsity improves the performance and stabilizes the training of neural networks.
INTRODUCTION
Many real-world datasets often contain data instances whose subset of input features is missing. While various imputing techniques, from imputing using global statisties such as mean, individually imputing by learning auxiliary models such GAN, can be applied with their own pros and cons, the most simple and natural way to do this is zero imputation, where we simply treat a missing feature as zero. In neural networks, at first glance, zero imputation can be thought of as reasonable solution since it simply drops missing input nodes by preventing the weights ussociated with them from being updated. Some what surprisingly, however, many previous studies have reported that this intuitive approach has an adverse effect on model performances (Hazan et al. 2015 Luo et al. 2018 Smieja al. 2018), and none of them has investigated the reasons of such performance degradations.
In this work, we find that zero imputation causes the output of neural network largely vary with respect to the number of missing entries in the input. We name this phenomenon Variable Sparsity Problem (VSP), which should be avoided in many real-world tasks. Consider movie recommender system, for instance. It is not desirable that users get different average of predicted ratings just because they have rated different number of movies (regardless of their actual rating values). One might argue that people with less ratings do not like movies in general and it is natural to give higher predicted values to people with more ratings. This might be partially true for users of some sparsity levels, but it is not common case uniformly applicable for a wider range of sparsity levels. This can be verified in real collaborative filtering datasets as shown in Figure T (upper left corner) where users have a similar average rating for test data regardless of the number of known ratings (see also other two examples in Fi. However, in standard neural networks with zero imputation, we observe that the model's is inference correlates with the number of known entries of the data instance shown in

---

Published as a conference paper at ICLR 2020
Collaborative Filtering
Electronic Medical Records
Single-cell RNA Sequence
5.0
0.25
15
4.5
0.20
mm
3.5
0.10
3.0
2.0
0.00
200
400
600
600
400
500
600
model prediction w/o SN
13.0
3.4
0.8
12.5
3.2
12.0
0.4
11.5
3.0
0.2
11.0
10
20
30
100
200
10
20
30
40
model predictinn SN
4.5
1.0
20.0
0.8
17.5
0.6
15.0
0.4
12.5
0.2
0.0
10
100
200
# known entries
Figure 1: First column: Sedhain et al. 20055) Movielens 100K (collaborative filtering) dataset. Second column: LSTM on Physionet 2012 (electronic medical records) dataset. Third column: Talwar et al. (2018) on Blakeley (single cell RNA sequence) dataset. First row: Mean of target values according to the number of known entries in training set. Second row: Predicted values of models with zero imputation according to the number of known entries for randomly selected test point. Input masks are randomly sampled (to artificially control its sparsity level). For each target sparsity level through x-axis, 50 samples are drawn, scattering the predicted values and plotting the average in solid line. Third row: Figures on how plots in second row are corrected by Sparsity Normalization.
the second row of Figure | It would be fatal in some safety- ceittcclii applications such as medical domain: a patient's probability of developing disease for example should not be evaluated differently depending on the number of medical tests they received (we do not want our model to predict the probability of death high just because some patient has been screened lot!).
In addition, we theoretically analyze the existence of VSP under several circumstances and propose simple yet effective means suppress VSP while retaining the intuitive advantages of zero imputation: normalizing with the number of non-zero entries for each data instance. We refer to this regularization Sparsity Normalization, and show that it effectively deals away with the VSP, resulting in significant improvements in both the performance and the stability of training neural networks.
Our contribution in this paper is threefold:
To best of our knowledge, we are the first in exploring the adverse effect of zero imputation, both theoretically and empirically.
Note that, this tendency is very consistent with other test points and is observed throughout the entire learning process (even before the training).
2

---

Published as a conference paper at ICLR 2020
(a) Neural network with variable sparsity levels
(b) with Sparsity Normalization
Figure 2: The change of output values according to the sparsity level of inputs (darker color indicates greater absolute value and dotted circles indicate missing nodes with zero imputation). SN makes the possible output range of a network be more stable with respect to the sparsity level.
We identify the cause of adverse effect of zero imputation, which we refer to variable sparsity problem, and formally describe how this problem actually affects training and inference neural networks (Section 2 We further provide new perspectives using VSP to understand phenomena that have not been clearly explained or that we have misunderstood (Section and
We present Sparsity Normalization (SN) and theoretically show that SN can solve the VSP under certain conditions (Section 33. We also experimentally reaffirm that simply applying SN can effectively alleviate or solve the VSP yielding significant performance gains (Section
2 VARIABLE SPARSITY PROBLEM
We formally define the Variable Sparsity Problem (VSP) as follows: phenomenon in which the expected value of the output layer of a neural network (over the weight and input distributions) depends on the sparsity (the number of zero values) of the input data (Figure 2a With VSP, the activation values of neural networks could become largely different for exactly the same input instance, depending on the number zero entries; this makes training more difficult and may mislead the model into incorrect predictions.
While zero imputation is intuitive in the sense that it drops the missing input features, will show that it causes variable sparsity problem for several example cases. Specifically, we show the VSP under assumptions with increasing generality: (Case 1) where activation function is an identity mapping with no bias, (Case 2) where activation function is an affine function, and (Case 3) where activation function is a non-decreasing convex function such as ReLU (Glorot et al. 2011). leaky
ReLU Maas et al. 2013), ELU (Clevert et al. 2016), or Softplus (Dugas et al. 2001).
Here, we summarize the notation for clarity. For L--laeer deep network with non-linearity a, we W E x denote the weight matrix of i-th layer, b' IR denote the bias, h' R denote the activation vector. For simplicity, use h R and h E R to denote input and output layer, respectively. Then, we have
h (-W'h-+++b" for i i
Our goal in this section is observe the change in h as the sparsity of ho (input x) changes. To simplify the discussion, consider the following assumption:
Assumption 1. (i) Every coordinate of input vector, f. is generated by the element-wise multipli- cation of two random variables it) and my where is binary mask indicating missing value and he is (possibly unobserved) feature value. Here, missing mask my MCAR (missing completely at random), with no dependency with other mask variables or their values ho All my follow some identical distribution with mean pms (ii) The elements of matrix W are mutually independent and follow the identical distribution with mean pdg Similarly, b' and To consist of i.i.d. coordinates with mean pi and paz respectively. (iii) pt is not zero uniformly over all i.
3

---

Published as a conference paper at ICLR 2020
(i) assumes the simplest missing mechanism. (ii) is similarly defined in Glorot & Bengio (2010) and He et al. (2015) in studying weight initialization techniques. (iii) may not hold under some initialization strategies, but as the learning progresses, it is very likely to hold.
(Case 1) For simplicity, let us first consider networks without the non non-linearity nor the bias term. Theorem shows that the average value of the output layer E[h is directly proportional to the expectation the mask vector pmm
Theorem 1. Suppose that activation a is an identity function and that bj is uniformly fixed as zero under Assumption Then, we have Elhf] II fi- Pm.
(Case 2) When the activation function is affine but now with a possibly nonzero bias, Elhf] is influenced by pm in the following way:
Theorem 2. Suppose that activation an affine function under Assumption 7 Suppose further that fa(a)) is defined a (nt-1)1 + pt)). Then, Fi fr. o fi (Ha//m))
(Case 3) Finally, when the activation function is non-linear but non-decreasing and convex, can show that [lff] is lower-bounded by some quantity involving jmm
Theorem 3. Suppose that o non-decreasing convex function under Assumption 7 Suppose further that f((x) is defined as a (nn-1p + 14] and pl > 0. Then, Elh/11 fL o...o fi (.u
If the expected value of the output layer (or the lower bound of it) depends on the level of spar- sity/missingness in Theorem [ even similar data instances may have different output values depending their sparsity levels, which would hinder fair and correct inference of the model. As shown in Figure I (second row), the VSP can easily occur even in practical settings of training neural networks where the above conditions do not hold.
SPARSITY NORMALIZATION
In this section, we propose a simple yet surpris- ingly effective method to resolve the VSP. We first revisit (Case 2) find a way of making expected output independent of input sparsity level since the linearity in activation simplifies the correction. Re- calling the notation of h = ho O (O represents the wise product), we find that simply nor- malizing via hgg (ho ( m) K1 Pm for any fixed constant Ki, can debias the dependency on
Algorithm Sparsity Normalization (SN)
Input: Dataset D, constant K. Output: Sparsity Normalized Dataset DsN.
Empty set S for each (h" .m)m E D do
hg K h [m
the input sparsity level. We name this simple nor-
S SUSUK { hgx)
malizing technique Sparsity Normalization (SN) and describe it in Algorithm]] I Conceptually, this method scales the size of each input value accord- ing its sparsity level so that the change in output
end for
DAN + S
size is less sensitive the sparsity level (Figure 2bp. The formal description correcting sparsity
bias by SN is as follows in this particular case:
Theorem 4. (With Sparsity Normalization) Suppose that activation is an affine function under Assumption 7 Suppose further that fa(x) = a(p4-1p + pb) and replace the input layer using SN i.e. h
m).Ka/ KK-KKpm (o any fixed constant Kj. Then, we have Elh41 = f K1).
Unlike Theorem 2 SN in Theorem 4 makes average activation to be independent of p's which determines the sparsity levels of input. It is not trivial to show the counterpart of (Case 3) using SN since [z)( o (E(r)) does not hold in general. However, we show through extensive experiments in Section 4 that SN is practically effective even in more general cases.
4

---

Published as a conference paper at ICLR 2020
Table I: Debiasing variable sparsity using SN on Moviclens datasets. Test RMSE with 95% confidence
interval 5-runs provided.
Datasets
Movielens 100K
Movielens IM
Movielens 10M
w/o SN
0.9346 = 0.0007
0.8831 0.0002
0.8859 0.0014
vector
AutoRec
w/ SN
0.9208 + 0.0023
0.8742 1 0.0003
0.8462 1 0.0005
Sedhain et 2015
w/o SN
0.8835 + 0.0003
0.8320
0.0003
0.7807
0.0017
item vector
w/ SN
0.8809 + 0.0011
0.8294 + 0.0004
0.7706 + 0.0023
w/o SN
0.9253 = 0,0010
0.8530
0.0006
0.8113 - 0.0058
user vector
CF-NADE
w/ SN
0.9231 = 0.0012
0.8525 + 0.0006
0.7854 = 0.0006
Zheng et al. 2016
w/o SN
0.8982 + 0.0005
0.8405
0.0007
N/A
item vector
w/ SN
0.8900 + 0.0018
0.8366
0.0008
N/A
w/o SN
0.8945 = 0.0024
0.8223 + 0.0016
N/A
CF-UleA (uu ct al. 2018
w/SN
0.8793 0.0017
0.8178 0.0007
N/A
Table 2: Comparison of AutoRec with SN and CF-UIcA with SN against state-of-the-arr collaborative filtering methods for each datasets. Bold font indicates neural network based models. The results marked f are taken from Zhang et al. (017)) and all other baselines are taken from original papers. The result of CF-UlcA with SN on Movielens 10M is not provided because the authors of CF-UlcA did not provide the results for it due to the complexity of the model.
Models
Movielens 100K
Models
Moviclens IM
Models
Movielens 10M
Koren 2008
0.913t
al. 2018
0
Sedlhain et al.
2015
0.782
Zhuang et al 2017
0.9114 0.0093
Loc ct a
2016
0,8333
Berg et al.
2018
0.777
0.911
2019
Dziugalte & Rov 2015
0,903
Berg al.
2018
0.832
Zheng al.
2016
0.771
Zhang ot at. 2017
0,901
Sedham et al.
2015
0.831
AutoKec w7 SN
(ours.)
0.7690 + 0.0023
et 2019
0,8889
Zhecng et al.
2016
0,829
Li 2016
0.7682 + 0.0003
0,8881
0.0017
AntoRec S
0,8260
0.0023
hin
0.7672 0.0001
AutoKec w/ SN (ours.)
0.8816 0.0087
Du el al. 2018]
0,823
cu mi
2013
0.766
CF-UIcA W/SN (ours.)
0.8779 0.0159
CF-UICA w/ SN (ours.)
0,8215
0.0037
Li al.
2017
0.7634 + 0.0002
While Theorem assumes that f'm is known and fixed across all data instances, we relax this assumption in practice and consider varying pmm across data instances. Specifically, by a maximum likelihood principle, we can estimate pmm for each instance by ||h" /no = ||m|| Thus, we In practice, we recommend using K as the average of [|mi|| over all instances in the training set. We could encounter the dying K can bring in regularization effect via controlling the magnitude of gradient (Salimans Kingma 2016). we define K mm) m [/1]] so that the average scales remain constant before and after the normalization, minimizing such side effects caused by SN.
have hΩN = K h²/ ||m||| where K- = Ro K1 (see Algorithm
ReLU phenomenon (He et al. 2015) if K too small (e.g., K = 1). Since the
4 EXPERIMENTS
In this section, we empirically show that VSP occurs in various machine learning tasks and can be alleviated by SN. In addition, we also show that resolving VSP leads to improved model performance on diverse scenarios.
4.1 COLLABORATIVE FILTERING (RECOMMENDATION) DATASETS
We identify VSP and the effect of SN on several popular benchmark datasets for collaborative filtering with extremely high missing rates. We train an AutoRec (Sedhain et al., 2015) using user vector on Movielens (Harper & Konstan 2016) 100K dataset for validating VSP and SN. Going back to the first column of Figure !| the prediction with SN is almost constant regardless of ||m ||2 Another perceived phenomenon in Figure D is that the higher the |m|] 1. the smaller the variation the prediction of
When [ | is 0, calculation is impossible. Hence, in this case, the ||m]| is assumed to be 1.
5

---

Published as a conference paper at ICLR 2020
Table 3: Debiasing variable sparsity using SN and comparisons against other missing handling techniques on five disease identification tasks of NHIS dataset. Test AUROC with 95% confidence interval of 5-runs is provided.
Dataset
Cardiovascular
Fatty Liver
Hypertension
Heart Failure
Diabetes
Zero Imputation w/o SN
0.7057 + 0.0027
0.6750 = 0.0050
0.7977 = 0.0027
0.7834 + 0.0036
0.9121
0.0097
Zero Imputation w/ SN (ours.)
0.7106 0.0005
0.6911 1 0.0022
0.8096 0.0010
0.7914 1 0.0012
0,9283
0.0611
Zero Impotation w/ BN
0.7033 = 0.0035
0.6691
0.0081
0.7828 = 0.0136
0.7749 + 0.0083
0,9026
0.0105
Zero Imputation w/ LN
0.7049 = 0.0035
0.0062
0.7944 0.0015
0.7820 0.0007
0.9127
0.0056
Dropout
0,7047 1 0.0021
0.6747
0.0069
0.7926 0.0038
0.7802 1 0.0049
0.9101
0.0054
Mean Imputation
0,7054 + 0.0025
0.6766
0.0067
0.7913 + 0.0019
0,7839 + 0.0021
0.9117
0.0075
Median Imputation
0.7056 + 0.0012
0.6713
0.0058
0.7865 + 0.0034
0.7818 + 0.0014
0.8975
0.0060
İ-NN
0.7052 1 0.0010
0.6874
0.0045
0,8000 :: 0.0050
0.7843 1 0.0024
0,9107 0.0075
MICE
0.7075 0.0022
0.0058
0.7957 0.0037
0.7875
0,9224
0.0021
SoftImpute
0.7014 + 0.0030
0.6868
0.0040
0.7990 0.0050
0,7828 + 0.0042
0.9224
0.0019
GMMC
0,7072 + 0.0016
0.6816
0.0067
0.7984
0.0062
0.7884 + 0.0M29
0.9109
0.0045
GAIN
0,7065 = 0.0015
0.6765
0.0060
0.7956 = 0.0020
0.7847 + 0.0031
0.9091 0.0067
the model with SN. Note that the same tendency has been observed regardless of test instances or datasets. This implies that models with SN yield more calibrated predictions; as more features are known for particular instance, the variance of prediction for that instance should decrease (since we generated independent masks in Figure [T. It is also worthwhile to note that an AutoRec is a sigmoid- based network and Movielens datasets are known to have no MCAR hypothesis (Wang et al.) 2018 in which Assumption I does not hold at all.
It is also validated that performance gains can be obtained by putting SN into AutoRec (Sedhain ett al. 2015). CF-NADE (Zheng et al.] 2016), and CF-UlcA (Du et al. 2018 which are the state-of- the-art among neural network based collaborative filtering models on several Movielens datasets (see Appendix B for detailed settings). In Table !
we consider three different sized Movielens datasets. Note that AutoRec and CF-NADE allow two types of models according to data encoding (user- or item-rating vector based) and we consider both types. While we obtain performance improvements with SN in most cases, it is more prominent in user-rating based model in case of AutoRec and CF-NADE.
Furthermore, Table compares our simply modification using SN on AutoRec and CF-UlcA with other state-of-the-ae collaborative filtering models beyond neural networks. Unlike experiments of AutoRec in Table [ which use the same network architectures proposed in original papers, here we could successfully learn more expressive network due to the stability obtained by using SNN For Movielens 100K and 1M datasets, applying SN yields better or similar performance compared to other state-of-the-ar collaborative filtering methods beyond neural network based models. It is important to note that all models outperforming AutoRec with SN on Movielens 10M are ensemble models while AutoRec with SN is a single model and it shows consistently competitive results across all datasets.
4.2 ELECTRONIC MEDICAL RECORDS (EMR) DATASETS
We further test VSP and SN for clinical time-series prediction with two Electronic Medical Records (EMR), namely PhysioNet Challenge 2012 (Silva et al.) 2012) and the National Health Insurance Service (NHIS) datasets, which have intrinsic missingness since patients will only receive medical examinations that are considered to be necessary. We identify whether the VSP exists with the PhysioNet Challenge 2012 dataset (Silva et al., 2012). We randomly select one test point and plot in-hospital death probabilities as the number of examinations varies (Second column of Figure []. Without SN, the ho-pisit death probability increases the number of examinations increases,
3 'e consider a CF-NADE without weight sharing and re-run experiments for fair comparisons because applying SN with weight sharing is not trivial. We also exclude averaging possible choices because it does not make big differences given unnecessary extra computational costs. *Because overfitting is less with SN in AutoRec, we twice the capacity than the existing AutoRec model.

---

Published as a conference paper at ICLR 2020
blakeley
kolodziejczyk
zeisel
8
25
RMSE
20
30
15
4-
10
2-
0.25
0.50
0.75
0.25
0.50
0.75
0.25
0.50
0.75
usoskin
quake
jurkat
15
12.5
10
RMSE
10.0
7.5
5.0
-5
0.25
0.50
0.75
0.25
0.50
0.75
0.25
0.50
0.75
Test Set Size (Ratio)
w/o SN
w/ SN
Figure 3: Debiasing variable sparsity using SN according test set ratio on six imputation tasks of single cell RNA sequence dataset. Test RMSE with 95% confidence interval of 10-runs is provided.
boston
diabetes
califomia
65
3.5
0.7
3.0
RISE
0.6
2.5
2.0
55
0.5
0.25
0.50
0.75
0.25
0.50
0.75
0.25
0.50
0.75
Dropout Prob.
W/o SN
w/ SN
Figure 4: Debiasing variable sparsity using SN with respect to drop rates on three popular UCI regression datasets. Test RMSE with 95% confidence interval of 5-runs is provided.
even though there is no such tendency in the dataset statistics. However, SN corrects this bias so that in-hospital death probability is consistent regardless of the number of examinations. We observe a similar tendency examples from the NHIS dataset as well.
Although SN corrects the VSP both datasets, we perceive different behaviors in both cases in terms of actual performance changes. While SN significantly outperforms its counterpart without SN on NHIS dataset shown in Table 3 it just performs similarly on PhysioNet dataset (results and detailed settings are deferred to Appendix C However, SN is still valuable for its ability to prevent biased predictions in this mission-critica area.
In addition, we compare SN with other missing handling techniques to show the efficacy of SN although our main purpose is to provide a deeper understanding and the corresponding solution about the issue that the zero imputation, which the simplest and most intuitive way of handling missing data, degrades the performance in training neural networks, Even with its simplicity, SN exhibits better or similar performances compared other more complex techniques. Detailed descriptions of other missing handling techniques are deferred to Appendix H

---

Published as a conference paper at ICLR 2020
90
w/o SN
w/ SN
88
w/ SN (all)
NL
86
84
82
2
4
6
8
# mask
Figure 5: Negative log likelihood of MADE on binarized MNIST with and without SN.
4.3 SINGLE-CELL RNA SEQUENCE DATASETS
Single-cell RNA sequence datasets contain expression levels of specific RNA for each single cells. Autolmpute (Talwar al., 2018) one of the -fate-of-the-- methods that imputes missing data on single-cell RNA sequence datasets. We reproduce their experiments using authors' official implemen- tation, and follow most of their experimental settings (see Appendix ( for details).
As before, first check whether VSP occurs in Autolmpute model. The third column in Figure | shows how the prediction of Autolmpute model changes as the number of known entries changes. Although the number of RNAs found in the specific cell is less related to cell characteristies (upper right corner in Figure [] the prediction increases as the number of RNAs found in the cell increases. This tendency is dramatically reduced with SN.
Figure 3 how imputation performance changes by being worked with SN to several single cell RNA sequence datasets with respect the portion of train set (see Appendix for more results). As we can see in Figure 3 SN significantly increases the imputation performance of Autolmpute model. In particular, the smaller the train data, the better the effect of SN in all datasets consistently. Autolmpute model is sigmoid- gease based function and single cell RNA datasets (Talwar al., 2018) do not have the MCAR hypothesis, unlike Assumption] | Nevertheless, VSP occurs even here and it can be successfully alleviated by SN with huge performance gain. It implies that SN would work for other neural network based imputation techniques.
4.4 DROPOUT ON UCI DATASETS
While SN primarily targets to fix the VSP in the input layer, it could be also applied to any layers of deep neural networks to resolve VSP. The typical example of having heterogeneous sparsity in hidden layers is when we use dropout (Srivastava et al. 2014), which can understood as another form of zero imputation but the hidden layers; with Bernoulli dropout, the variance of the number of zero units across instances rpp(1 - p) (n:: the dimension of hidden layer, p: drop rate). While dropout partially handles this VSP issue by scaling 1/(1 p) in the training phase" SN can exactly correct VSP of hidden layers by considering individual level sparsity (Note that the scaling of dropout can be viewed as applying SN in an average sense: E m|]1 ] re(1 -pppp) p) and K n).
'In almost all of the deep learning frameworks such as PyTorch, TensorFlow, Theano and Caffe, this inverted dropout is supported.
8

---

Published as a conference paper at ICLR 2020
Figure 4 shows how the RMSE changes the drop rate changes with and without SN on three popular UCI regression datasets (Boston Housing, Diabetes, and California Housing 6 As illustrated in Figure the larger the drop rate, the greater the difference of RMSE between with and without SN. To explain this phenomenon, we define the degree of VSP as the inverse of sigmal-to-noise ratio with respect to the number of active units in hidden layers: v // (n\1 - p)) (expected number of active units over its standard deviation). As can seen from the figure, the larger drop rate is, the more severe degree of VSP is and thus the greater protection by SN against performance degradation.
4.5 DENSITY ESTIMATION
In the current literature of estimating density based on deep models, imputs with missing features are in general not largely considered. However, we still may experience the VSP since the proportion of zeros in the data itself can vary greatly from instance to instance. In this experiment, we apply SN to MADE (Germain al. 2015), which is one of the modern architectures in neural network-based density estimation. We reproduce binarized MNIST (LeCun (998) experiments of Germann et al.) (0133) measuring negative log likelihood (the lower the better) of test dataset while increasing the number of masks. Figure illustrates the effect of using SN. Note that MADE uses masks in the hidden layer that are designed produce proper autoregressive conditional probabilities and variable sparsity arises across hidden nodes. SN can be trivially extended to handle this case well and the corresponding result is given in the figure denoted as w/SN (al1) We reaffirm that SN is effective even when MCAR assumption is not established.
RELATED WORKS
Missing handling techniques Missing imputation can be understood as a technique to increase the generalization performance by injecting plausible noise into data. Noise injection using global statisties like mean or median values is the simplest way to do this (Lipton al., 2016 Smieja al. 2018). However, it could lead to highly incorrect estimation since they do take into consideration the characteristics of each data instance (Tresp al. 1994 Che et al. 2018 To overcome this limitation, researchers have proposed various ways to model individualized noise using autoencoders (Pathak et al. 2016; Gondara & Wang 2018), or GANs (Yoon et al. 2018 Li ct al. 2019). However, those model based imputation techniques have not properly worked for high dimensional datasets with the large number of features and/or extremely high missing rates (Yoon) et al. 2018 because excessive noise can ruin the training of neural networks rather increasing generalization performance.
For this reason, in the case of high dimensional datasets such as collaborative filtering or single cell RNA sequences, different methods of handling missing data have been proposed. A line of work simply used zero imputation by minimizing noise level and achieved state-of-the-ar performance on their target datasets (Sedhain et al. 2015 Zheng et al. 2016 Talwar et al. 2018). In addition, methods using low-rank matrix factorization have been proposed to reduce the input dimension, but these methods not only cause lots of information loss but also fail to capture non- -oneaiiy of the input data (Hazan et al. 2015; Bachman et al.) 2017 He et al.) 2017). Vinyals et al. (2016); Monti ct al. (2017) proposed recurrent neural network (RNN) based methods but computational costs for these methods are outrageous for high dimensional datasets. Also, it is not natural to use RNN-based models for non- oon-sequentall datasets.
'The most experimental settings are adopted from Klambauer et al. 2017 Littwin & Wolf 2018 UCI experiments (see Appendix for for detail).
"hhe detailed description of the extension deferred Appendix

---

Published as a conference paper at ICLR 2020
Other forms of Sparsity Normalization We discuss other forms of SN to alleviate VSP, already in use unwitringly due to empirical performance improvements. DropBlock (Ghiasi et al. 2018) compensates activation from dropped features by exactly counting mask vector similar to SN (similar approach discussed in Section 4.4) It is remarkable that we can find models using SN-like normal- ization even in handling datasets without missing features. For example, in CBOW model (Mikoloy ct al., 2013) where the number of words used as an input depends on the position the sentence, was later revealed that SN like normalization has a practical performance improvement. As an another example, Kipf Welling (2017) applied Laplacian normalization which is the standard way of representing graph in graph theory, can naturally handle heterogeneous node degrees and precisely matches the SN operation. In this paper, we explicitly extend SN, which was limited and unintentionally applied to only few settings, to a model agnostic technique.
6 CONCLUSION
We identified variable sparsity problem (VSP) caused by zero imputation that has not been explicitly studied before. To best our knowledge, this paper provided the first theoretical analysis on why zero imputation is harmful to inference of neural networks. We showed that variable sparsity problem actually exists in diverse real-world datasets. We also confirmed that theoretically inspired normalizing method, Sparsity Normalization, not only reduces the VSP but also improves the generalization performance and stability of feed-forwarding of neural networks with missing values, even in areas where existing missing imputation techniques do not cover well (e.g., collaborative filtering, single cell RNA datasets).
ACKNOWLEDGMENTS
This work was supported the Institute of Information & Communications Technology Planning & Evaluation (ITP) grants (No. 16.2001----0566,, No.2017-0-01779 and No.2019-0-0137I, the Na- tional Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) grants (N..2018RIASAA0520
and 222001RICICI109110 bation Center of Samsung Electronics via SRFC-ITI702-15. the National IT Industry Promotion Agency grant funded by the Ministry of Science and ICT, and the Ministry of Health and Welfare (NO. S0310-199 10-10000,,, Development Project of The Precision Medicine Hospital Information System (P-HIS)).
the Samsung Research Funding & Incu-
REFERENCES
Philip Bachman, Alessandro Sordoni, and Adam Trischler. Learning algorithms for active learning. In Proceedings of the 34th International Conference Machine Learning Volume 70, pp. 301-310. JMLR. org, 2017.
Rianne van den Berg, Thomas N Kipf, and Max Welling. Graph convolutional matrix completion. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining Deep Learning Day. ACM, 2018.
S van Buuren and Karin Groothuis-Oudshoomm mice: Multivariate imputation by chained equations in r. Journal of statistical software, pp. 1-68, 2010.
Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. Brits: Bidirectional recurrent imputation for time series. In Advances in Neural Information Processing Systems, pp. 6775-6785, 2018.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):6085, 2018.
10

---

Published as a conference paper at ICLR 2020
Chao Chen, Dongsheng Li, Qin Lv, Junchi Yan, Stephen M Chu, and Li Shang. Mpma: Mixture probabilistic matrix approximation for collaborative filtering. In IJCA,, pp. 1382-1388, 2016.
Chao Chen, Dongsheng Li, Qin Lv, Junchi Yan, Li Shang, and Stephen M Chu. Gloma: Embedding global information in local matrix approximation models for collaborative filtering. Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Djork-Amé Clevert, Thomas Unterthiner. and Sepp Hochreiter. Fast and accurate deep network learn- ing by exponential linear units (elus). In International Conference on Learning Representations, 2016.
Chao Du, Chongxuan Li, Yin Zheng, Jun Zhu, and Bo Zhang. Collaborative filtering with user-item co-autoregressive models. In Thirty-Secone AAAI Conference on Artificial Intelligence, 2018.
Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia. Incorporating second-order functional knowledge for better option pricing. In Advances in neural information 2001.
processing systems, pp.
Gintare Karolina Dziugaite and Daniel M Roy. Neural network matrix factorization. arXiv preprint arXiv:1511.06443, 2015.
Mingsheng Fu, Hong Qu Dagmawi Moges, and Li Lu. Attention based collaborative filtering. Neurocomputing, 311:88-98, 2018.
Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation. In International Conference on Machine Learning, pp. 881-889, 2015.
Golnaz Ghiasi, Tsung- Yi Lin, and Quoe V Le. Dropblock: regularization method for convolutional networks. In Advances in Neural Information Processing Systems, pp. 10727-10737, 2018.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp- 315-323, 2011.
Lovedeep Gondara and Ke Wang. Mida: Multiple imputation using denoising autoencoders. In Pacific-Asia Conference Knowledge Discovery and Data Mining, pp. 260-272. Springer, 2018.
F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis) 5(4):19, 2016.
Elad Hazan, Roi Livni, and Yishay Mansour. Classification with low rank and missing data. In Proceedings of the 32nd International Conference on Machine Leurning- Volume 68, pp. 257-266. JMLR. org, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In Proceedings of the 26th International Conference on World Wide Web, pp. 173-182. International World Wide Web Conferences Steering Committee, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint 152..0350 2015.

---

Published as a conference paper at ICLR 2020
Thomas N Kipf and Max Welling. Semi- Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. Advances in neural information processing systems, pp. 971-980, 2017.
Yehuda Koren. Factorization meets the neighborhood: multifaceted collaborative filtering model. In Proceedings of the 14th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 426-434. ACM, 2008.
Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer, pp. 30-37, 2009.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mmist// 1998.
Joonseok Lee, Seungyeon Kim, Guy Lebanon, Yoram Singer, and Samy Bengio. Llorma: Local low-rank matrix approximation. The Journal of Machine Learning Research, 17(1):4424465, 2016.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXivv1607.064500 2016.
Dongsheng Li, Chao Chen, Qin Lv, Junchi Yan, Li Shang, and Stephen Chu. Low-rank matrix approximation with stability. International Conference on Machine Learning, pp. 295-303, 2016.
Dongsheng Li, Chao Chen, Wei Liu, Tun Lu, Ning Gu, and Stephen Chu. Mixture-rank matrix approximation for collaborative filtering. In Advances in Neural Information Processing Systems, pp. 477-485, 2017.
Steven Cheng-Xian Xian Li, Bo Jiang, and Benjamin Marlin. Misgan: Learning from incomplete data with generative adversarial networks. arXiv preprint vrXiv:/902.095900 2019.
Zachary C Lipton, David C Kale, and Randall Wetzel. Modeling missing data in clinical time series with mns. Machine Learning for Healthcare, 2016.
Etai Littwin and Lior Wolf. Regularizing by the variance of the activations' sample-variances In Advances in Neural Information Processing Systems, pp. 2115-2125, 2018.
Yonghong Luo, Xiangrui Cai, Ying Zhang, Jun Xu, al. Multivariate time series imputation with generative adversarial networks. In Advances in Neural Information Processing Systems, pp. 1607, 2018.
Andrew L Maas, Awni Y Hannun, and Andrew Ng. Rectifier nonlinearities improve neural network acoustic models. Proc. icml, volume 30, pp. 3, 2013.
Rahul Mazumder, Trevor Hastie, and Robert Tibshirani. Spectral regularization algorithms for learning large incomplete matrices. Journal of machine learning research, 1(A2u))2287-2322 2010.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation word representa- tions in vector space. In International Conference on Learning Representations, 2013.
Federico Monti, Michael Bronstein, and Xavier Bresson. Geometric matrix completion with recurrent multi-graph neural networks. In Advances in Neural Information Processing Systems, pp. 3697- 3707, 2017.
Deepak Pathak, Philipp Krahenbuhi, Jeff Donahue, Trevor Darrell, and Alexei Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536-2544, 2016.
12

---

Published as a conference paper at ICLR 2020
Ruslan Salakhutdinov, Andriy Mnih, and Geoffrey Hinton. Restricted boltzmann machines for collaborative filtering. In Proceedings of the 24th international conference Machine learning, pp. 791-798. ACM, 2007.
Tim Salimans and Durk Kingma. Weight normalization: simple reparameterization accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pp. 901-909, 2016.
Suvash Sedhain, Aditya Krishna Menon, Scott Sanner, and Lexing Xie. Autorec: Autoencoders meet collaborative filtering. In Proceedings of the 24th International Conference on World Wide Web, pp. 111-112. ACM, 2015.
I. Silva, George B. Moody, Daniel J. Scott, L. A. Celi, and R. Gritz Mark. Predicting in-hospital mortality of icu patients: The physionet/comppting in cardiology challenge 2012. 2012 Computing in Cardiology, pp. 245-248. 2012.
Marek Smieja, Lukasz Struski, Jacek Tabor, Bartosz Zielińskii, and Przemystaw Spurek. Processing of missing data by neural networks. In Advances in Neural Information Processing Systems, pp. 2719-2729, 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-1958 2014.
Divyanshu Talwar, Aanchal Mongia, Debarka Sengupta, and Angshul Majumdar. Autoimpute: Autoencoder based imputation of single-cell rnasseq data. Scientific reports, 8(1):16329, 2018.
Volker Tresp, Subutai Ahmad, and Ralph Neuneier. Training neural networks with deficient data. In Advances in neural information processing systems, pp. 125-13,.1 1994.
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. Order matters: Sequence to sequence for sets. In International Conference on Learning Representations, 2016.
Menghan Wang, Mingming Gong, Xiaolin Zheng, and Kun Zhang. Modeling dynamic missingness of implicit feedback for recommendation. In Advances in neural information processing systems, pp. 6669-6678, 2018.
Baolin Yi, Xiaoxuan Shen, Hai Liu, Zhaoli Zhang, Wei Zhang, Sannyuya Liu, and Naixue Xiong. Deep matrix factorization with implicit feedback embedding for recommendation system. IEEE Transactions on Industrial Informatics, 2019.
Jinsung Yoon, James Jordon, and Mihaela Van Der Schaar. Gain: Missing data imputation using generative adversarial nets. In Proceedings of the 35th International Conference Machine Volume 71, 2018.
Shuai Zhang, Lina Yao, and Xiwei Xu. Autosvd++: An efficient hybrid collaborative filtering model via contractive auto-encoders. In Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval, pp. 957-960. ACM, 2017.
Yin Zheng, Bangsheng Tang, Wenkui Ding, and Hanning Zhou. A neural autoregressive approach to collaborative filtering. In Proceedings of the 33rd International Conference on Machine Learning- Volume 69, JMLR. org, 2016.
Fuzhen Zhuang, Zhiqiang Zhang, Mingda Qian, Chuan Shi, Xing Xie, and Qing He. Representation learning via dual autonnoroee for recommendation. Neural Networks, 90:83 89, 2017.
13

---

Published as a conference paper at ICLR 2020
A PROOFS
A.1 PROOF OF THEOREM
Proof. From the definition of hu, wt, h1 Ni. my, the following equation holds.
E = no El hi] nu mm] From the Assumption wt. h.. and my are independent of each other. Thus,
Eiil no Bl [0
Similarly, the following holds.
Ebhil Ew for ......1
Since h h and wi are independent of each other by the Assumption I and the definition of h
EThi] = n [ui 'J. Therefore.
B14 II mi Eluof Emm 1=1
II ==1
ne- 1 Ma
A.2 PROOF OF THEOREM 2
Proof. From the definition of hy, wt hi,hh my and the property of an affine function (BIl))
El)( the following equation holds.
E Eb4 (nn Elw 4P] B(H)
-no Eiot Afmm + Elb!))
From the Assumption w/. h.. and my are independent of each other. Thus,
Eih/1 PePm + mb)
| + Elb!1)
fa (Hattm)
Similarly, the following holds.
Elhi] (n, El Eluth
ETbil) for i=
Since h1-1 and wf are independent of each other by the Assumption I and the definition of h/-1
Elh (n--1 El E(ii)
Therefore,
Elil = fL o...o f2(2pmm)
14

---

Published as a conference paper at ICLR 2020
A.3 PROOF OF THEOREM
Proof. From the definition of fu, wi hf. hi, my and the property of convex function Blo(-) 2 a (F(-)), the following equation holds.
E (no hi] E(b() nno [w] mfmm + E76!l)
From the Assumption w/. ng. and my are independent of each other. Thus,
EM 2 Eno ([tg] [(mi] + E(bH1)
(nopl mb)
= fi (HaHmm)
Similarly, the following holds.
E[M] Fluth + E(bil) for 1o.==1,.. 1,...
Since hy and wnf are independent of each other by the Assumption I and the definition of hj
EThi] 2 E|h +4))
E(bi)
Since we assume that a is non-decreasing, we finally get
Eli 2 ft .... f1(12Mm)
A.4 PROOF OF THROREM 4
Proof. By theorem 2 Fi fr (F((g)))) Since F. F = FF[ho]- K/ pms
Ehill fr f K/um) fL o f K)
15

---

Published as a conference paper at ICLR 2020
B COLLABORATIVE FILTERING (RECOMMENDATION) DATASETS
B.1 DETAILED EXPERIMENTAL SETTINGS OF TABLE
This subsection describes the experimental settings of detailed collaborative filtering tasks in Section 4 As already mentioned, we follow the settings of AutoRec, CF-NADE, and CF-UlcA as much as possible. We perform each experiments by 5 times, and report mean and 95% confidence intervals. We randomly select 10% of the ratings of each datasets for the test set (Harper & Konstan 2016). As the dataset is too small for Movielens 100K and IM datasets, the confidence interval tends to be large by changing the dataset split. Hence, the same dataset split is used in each 5 experiments Table
AutoRec (Sedhain et al. 2015) We use two layer AutoRec model with 500 hidden units. For fair comparisons, we tune the ayper-parameters for weight decay in all experiments to have only one significant digit, and use learning rate of 10-3 except on Movielens 10M where we use learning rate of 10 4. We use full batch on Movielens 100K and IM, mini-batch (1000) on Movielens 10M. Besides, we use Adam optimizer instead of Resilient Propagation (RProp) unlike the AutoRec paper. The RProp optimizer shows fast convergence but can only be used in full batch scenario. It is not possible for 12GB of GPU memory to use full batch in training large dataset such as Movielens 10M. Thus, we decide to use Adam optimizer rather than RProp optimizer. Fortunately, although the optimizer is changed to Adam, the prediction performance is not degraded in most cases. The experimental results of comparing both optimizers are summarized in Table
Table 4: Comparison of Test RMSE between Adam and Resilient Propagation optimizer on Movielens 100K, 1M, and 10M datasets.
Datasets
Movielens 100K
Movielens 1M
Movielens 10M
imput vector
item vector
user vector
item vector
user vector
item vector
user vector
RProp Adam
0.8861
0.9437
0.8358
0.8804
0.7821
0.867
0.8831
0.9343
0.8306
0.8832
0.7807
0,8859
1: Taken from Sedhain et al.
2015
CF-NADE (Zheng et al., 2016) We use two layer CF-NADE model with 500 hidden units. For fair comparisons, we tune the hyper-parameters for weight decay in all experiments to have only one significant digit, and use learning rate @ of 0.001. Also, use mini-batch (512) just following CF-NADE. Although the CF-NADE used weight sharing and averaging possible choices in addition to weight decay, report the results without weight sharing and averaging possible choices in Table because it is not clear how to apply SN with weight sharing, and there is almost no performance gain with averaging possible choices despite of its high computational costs. Furthermore. we do not experiment on Movielens 10M with item vector encoding because the authors of CF-NADE did not provide the results for due to the complexity of the model.
2 CF-UTCA (Du et al., 2018 We use the authors' official code and the train/test dataset splits for these experiments. Since CF-UlcA is a model that accepts both user and item vector as input, is not necessary to consider two types of encoding as in AutoRec or CF-NADE. On the other hand, reasonable take different K values for user and item vector with SN. We treat K as 66 for the user vector and 110 for the item vector. For models without SN, 0.0001 used the parameter for weight decay as suggested by the CF-UlcA. It is natural use different À values for SN because the optimal À must be changed along with SN. Hence, we use X = 0.0005 for Movielens 100K and 0.00006 for Movielens 1M in the case of SN. As in CF-NADE, we do not test Movielens 10M because the authors of CF-UICA also did not report for it owing to its high computational cost.
'The CF-NADE uses learning rate 5 X 1 for Moviclens 10M, but use 10 for fast convergence. Therefore, the results can be somewhat different from original paper. https: thub. com tttu//tummmmmmm m /CF-UICA
16

---

Published as a conference paper at ICLR 2020
B.2 DETAILED EXPERIMENTAL SETTINGS OF TABLE
In comparison with other sfate-of-the art models, used 1000 hidden units in AutoRec (Sedhain et al. 2015) with SN. While Sedhain et al. (2015) claimed that they were able to achieve enough performance only with 500 hidden units, 500 hidden units did not achieve sufficient performance when applying SN. The Figure plots the test RMSE, changing the number of hidden units for Movielens 100K and IM. We can see that 600 units for Movielens 100K and 900 units for Movielens IM are necessary for getting better performance. Obviously, as datasets become more complex and larger, we need more network capacity. Therefore, we decide to use two times larger network capacity (1000 hidden units) to get better performance. The number of hidden units can also be viewed and tuned hyper-parameter, and have not tuned much for the number of hidden units. Note that, unlike Table [ report the results with five random splits for all datasets in Table to compare fairly with other state-of-the-an methods.
0.62925
5.82400
Twa
0.800
5.82909
5.82775
0.8000
2.52750
400
300
003
700
802
500
1000
500
000
700
500
200
1000
Niddes uniss
hidee units
Figure 6: Test RMSE AutoRec with SN on Movielens 100K (left) and Movielens 1M (right) with respect the number hidden units.
17

---

Published as a conference paper at ICLR 2020
C ELECTRONIC MEDICAL RECORDS (EMR) DATASETS
C.I NHIS DATASET
Table 5: Debiasing variable sparsity in the case of applying dropout using SN on five disease identification tasks of NHIS dataset. Test AUROC with 95% confidence interval of S-runs is provided.
Dataset
Curdiovascular
Fany Liver
Hypertension
Heart Failure
Diabetes
Zero Imputation w/o SN Imputation (ours.)
0.7084 + 0,0005
0.6858 + 0.0065
0.8023 = 0.0054
0,7876 0.0012
0.9263 0.0026
0.0009
0.6941
0.8086
0.7922
0.0029
The NHIS dataset, which is from National Health Insurance Service (NHIS), consists of medical diagnosis of around 300,000 people. The goal is to predict the occurrence of 5 diseases. Each patients takes 34 examinations over 5 years. We split dataset into two set (train and test). where the ratio of train and test split is 3:1. We pre-process input data with min-max normalization, which makes min and max values of each features be zero and one following GAIN (Yoon et al.) (018). We train 2 layer neural networks which have 50 and 30 hidden units each, and evaluate the model with AUROC. We use ReLU activation, and dropout rate as 0.8 (if applied). Since these dataset too imbalance, we apply class weight to the loss function for handling label imbalance. Besides, we use Adam optimizer with learning rate 10-2 without weight decay, and full batch. We evaluate the model on times and report mean and 95% confidence interval. We also observe that Sparsity Normalization makes performance gain even when the dropout is integrated in the networks (See Table 5).
Data source This study used the National Health Insurance System-National Health Screening Cohort (NHIS-HEALS)* data derived from national health screening program and the national health insurance claim database in the National Health Insurance System (NHIS) of South Korea. Data from the NHIS-HEAL. 10 was fully anonymized for all analyses and informed consent was not specifically obtained from each participant. This study was approved and exempt from informed consent by the Institutional Review Board of Yonsei University, Severance Hospital in Seoul, South Korea (IRB (o.4-2016-0383))
Data Availability Data cannot be shared publicly because of the provisions of the National Health Insurance Service (NHIS). Korean legal restrictions prohibit authors from making the data publicly available, and the authority implemented the restrictions is NHIS (National Health Insurance Service), one of the govermment agency of Republic of Korea. NHIS provides limited portion of anonymized data to the researchers for the purpose of the public interest. However, they exclusively provide data to whom made direct contact of the NHIS and agreed to policies of NHIS. Redistribution of the data not permitted for the researchers. The contact name and the information to which the data request can be sent: Haeryoung Park Information analysis department Big data operation room NHISS Tel: + +82- 002--3-733 2430. E-mail: 1 Lumen 7 70hniss............... or .r.
"SSenn SC, Kim YY. Park SK, et al. Cohort profile: the National Health Insurance Service-National Health
Screening Cohort (NHIS-HEAL.S) in Korea. BMJ Open 2017;7:e016664 6640, pmid:28947447.
18

---

Published as a conference paper at ICLR 2020
C.2 PHYSIONET CHALLENGE 2012 DATASET
Table 6: Debiasing variable sparsity using SN mortality prediction task of PhysioNet Challenge 2012. Test AUROC with 95% confidence interval of 5-runs provided.
Model
Test AUROC
Zero Imputation w/o SN Zero Imputation SN (ours.)
0.8177 = 0.0071
0.8152 0.0010
PhysioNet Challenge 2012 dataset Silva et al. 2012) consists of 48 hours-long multivariate clinical time series data from intensive care unit (ICU). Most of the experimental settings are followed by BRITS (Cao et al. 2018). We divide 48 hours into 288 timesteps, which contain 35 examinations each. The goal this task to predict in-hospital death. We use dataset split given by PhysioNet Challenge 2012 (Silva al., 2012) where each of them contains 4000 data points. In the preprocessing phase, and standard deviation and fill zero for missing values (zero imputation). We train single layer LSTM network which has 108 hidden units, and evaluate the model with AUROC. We apply class weight to handle imbalance problem in the dataset like in setting above. We use Adam optimizer with learning rate 2 x 10-1 512 batch size, and early stopping method based on AUROC of validation set. We evaluate the model on times and report mean and 95% confidence interval. Note that input of LSTM model is the results for 35 medical examinations at specific timestamp. Hence, we apply SN separately for each timestamp. As the aforementioned state of Section 4.2 SN could not make significant performance gain in PhysioNet Challenge 2012 dataset (Silva al. 2012) though Sparsity Normalization eases the VSP (See T Nevertheless, SN is still valuable for its ability prevent biased predictions in this mission-critican area.
standardize (make mean
for each features) the input features
19

---

Published as a conference paper at ICLR 2020
D SINGLE-CELL RNA SEQUENCE DATASETS
In the Autolmpute experiments, run the experiments using the author's public code Talwar et al. (2018) reported experimental results on eight datasets (Blakeley, Jurkat, Kolodziejezyk, Preimplan- lation, Quake, Usoskin, Zeisel, and PBMC). Since we can obtain preprocessed datasets only for seven datasets except PBMC, we run experiments on seven single cell RNA sequence datasets All experimental settings without SN are exactly followed by the author's ' code and the parameer settings published in Table 2 of their paper. For the model integrated with SN, all the experimental settings are followed by the original model except that the smaller threshold for early stopping is taken because the models with SN tends to underfitted by using the threshold as suggested by the authors 13 In addition, Talwar al. (008)) conducted experiments only cases with test ratios of 10.1, 0.2, 0.3, 0.4, 0.5), but explored more test set ratios {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9) to show that the SN works well under more sparsity (extremely high test set ratio). Like the authors' settings, we perform 10 experiments and report mean and 95% confidence intervals. We change the test set split on every trials. It is remarkable that although hyper-parameten are not favorable for SN, SN performs similarly or better on all seven datasets that we concerned (See Figure 3 and
preimplantation
RMSE
-5
0.2
0.8 Test Set Size (Ratio)
0.4
0.6
w/o SN
w/ SN
ratio imputation task of Preimplantation dataset. Test RMSE with 95% confidence interval of 10-runs is provided.
Figure 7: Debiasing variable sparsity using SN according
E DROPOUT ON UCI DATASETS
Most experimental settings are adopted from Klambauer al. (2017 Litwii & Woll (2018) UCI experiments. We use ReLU (Glorot al. 201 networks of hidden layers with 256 units. We Mcan Square Error (MSE) for loss function and Adam optimizer without weight decay, learning rate 10-4, The batch size used for training is 128. We split 10% from the dataset for test and use 20% of training set as the validation set. For all inputs, we applied min- max normalization, which makes min and max values of each features be 0 and 1. Note that, we use dense datasets without any missing attributes to focus on the effects of dropout. All the datasets are used as provided the package in sklearn. dat asets.
10-8 and
ykl
We slightly tune the hyper-parameter À of weight decay only for the Preimplantation dataset with SN
20).
20

---

Published as a conference paper at ICLR 2020
F DENSITY ESTIMATION
To reproduce binarized MNIST experiments of MADE Germain al. 2015). we adopt and slightly modify public implementation of MADE Figure shows our reproduction results of Figure in the original paper. We follow most settings of the MADE. We use single hidden layer MADE networks with 500 units, learning rate 0.005, and test on authors' binarized MNIST datase 15 The only difference from the original authors' implementation is that we used Adam (c = 10 and no weight decay), rather using Adagrad. Because observed that the model is undefitted with Adagrad while the learning rates of each element of weight matrix is dropped so rapidly.
Since there is missingness in the Binarized MNIST dataset, it cannot be divided by [|]]] as suggested by Algorithm In this experiment, generality. That is, all pixels that are 0 in the binarized MNIST dataset are regarded missing. We label results of this with w/SN SN and plot them in Figure 5 As the aforementioned state in Section 4.5 MADE can also cause variable sparsity by mask matrices of each weight. In MADE, the connection between specific units is forcibly controlled through a mechanism of element-wise product of a specific mask matrix M in weight W E Ih X These mask matrices also cause variation in sparsity. We plot the results using the new mask matrix MSN in place of the mask matrix M' used by MADE with w/SN (all) in Figure The method of calculating the new mask matrix M using the existing mask matrix M is follows:
regard || as ||h
not lose the
Més + (1" (1'M)) M @ (M'11")
where 2 denotes element-wisc division and is a column vector where all elements are (1" 1/1/).) corresponds to and (M'11")) does to |mm]] Algorithm
We demonstrate the effectiveness of SN even in situations where the learning rate is smaller. Smaller learning rate (0.001) shows the effect of SN more clearly despite of their longer training time show in Figure g
SN w/ SN
88
NLL
84
80
2
Figure 8: Negative log- likelihoddo of MADE on binarized MNIST with and without SN in the case of small learning rate (0.001).
G MACHINE DESCRIPTION
We perform all the experiments on Titan X with 12GB of VRAM. 12 GB of VRAM is not always necessary, and most experiments require smaller VRAM.
ottees/s/ggtthiooookkrrtts//ssrr mist.npp
21

---

Published as a conference paper at ICLR 2020
H COMPARISON TO OTHER MISSING HANDLING TECHNIQUES
In this section, we compare Sparsity Normalization with other missing handling techniques. Although the main contribution of our paper is to provide a deeper understanding and the corresponding solution about the issue that the zero imputation, the simplest and most intuitive way of handling missing data, degrades the performance in training neural networks. Nonetheless, we show that Sparsity Normalization effective for high dimensional datasets (with a large number of features and high missing rates) via collaborative filtering dataset, while showing that Sparsity Normalization has competitive results with other modern missing handling techniques for datasets with non-high dimensional setting (electronic medical records datasets, UCI datasets w/ and w/o MCAR assumption).
For fair comparisons, consider the tasks in Section 4 as well as the tasks used by modern missing handling techniques such as GAIN (Yoon et al. 201 8) and Smieea et al. (0018).
As baselines, we consider modern missing handling techniques such as GAIN, Smieja et al. 2018 and their baselines as well.
Zero Imputation w/o Sparsity Normalization: Missing values are replaced with zero.
Zero Imputation w/ Sparsity Normalization (ours.): Based on zero imputation, apply Sparsity Normalization (SN).
Zero Imputation with Batch Normalization (loffe & Szegedy 2015): Based on zero imputation, apply Batch Normalization (BN) only on the first layer.
Zero Imputation with Layer Normalization (Lei Ba et al. 2016): Based zero imputation, apply
Layer Normalization (LN) only on the first layer.
Missing values are replaced with zero and other values are divided by (E(r,, mm)ED m|ll]/no) like standard dropout (Srivastava et al. 2014). Dropout uses sin- gle missing (drop) probability uniformly across all instances of the dataset while SN normalizes each data instance with its own missing rate.
Dropou 16
Mean Imputation: Missing values are replaced with the mean of those features.
Median Imputation: Missing values are replaced with the median of those features.
k-Nearest Neighbors (k-NN): Missing values are replaced with the mean of those features from
k nearest neighbor samples. We use = 5 following Smieja et al. (2018) 's experimental setting.
Multivariate Imputation by Chained Equations (MICE): Proposed by Buuren & Groothuis Oudshoorn 2010).
SoftImpute Mazumder et al. 2010
Gaussian Mixture Model Compensator (GMMC): Proposed by Smieja et al. (018 In the case of GMMC, any activation functions except ReLU and RBF (Radial Basis Function) is prohibitive on the first hidden layer. Thus, the activation function of the first hidden layer is replaced by ReLU in all base architectures without ReLU,
GAIN Yoon al. 2018
We implement Mean Imputation, Median Imputation, MICE, k-NN and SoftImpute by python package Jancyimputes We use authors' official codes for GMMC [8] and GAIN T9) Layer Normalization and Batch Normalization are not commonly considered in studies of handling missing data. However, we additionally take these as baselines because they have similarities to Sparsity Normalization in terms of stabilizing the statistics of a hidden laye 20 (See Appendix H.2 for deeper analysis).
The GMMC (Smieja et al. 2018) used this method as their baseline.
VIn Yoon et al. (0018) this method was named Matrix.
networkss
com http://w...ooo//yyoooooooooooo// jsyoon0 323/CAIN /CAIN
We only consider LN and BN in the of applying the first hidden layer. Because find that the prediction performance is more worse when LN or BN applied to all the hidden layers, and it is difficult to fairly compare with the Sparsity Normalization.
22

---

Published as a conference paper at ICLR 2020
H.1 COLLABORATIVE FILTERING (RECOMMENDATION) DATASET
In this section, compare the SN with other missing handling techniques using the collaborative filtering dataset. Appendix H.I.1 compares the prediction performance of the bascline methods and the SN, while Appendix H.1 1.2 deeply analyzes the characteristics of the SN in comparison with Layer Normalization (Lei Ba et al. 2016) and Batch Normalization (loffe & Szegedy (015).
H.1.1 COMPARISON OF PREDICTION PERFORMANCE
is considered training an AutoRec (Sedhain et al. 2013) model on the Movielens 100K dataset. Most experimental settings are adopted from Section 4. We evaluate each missing handling techniques on both data encoding (user- or item-rating vector) as shown in Table both encoding. Sparsity Normalization performs better similar other missing handling techniques. While some missing handling techniques perform poorly rather than zero imputation depending on the encoding. Sparsity Normalization improves performance consistently for both data encodings. It is worth mentioning that Sparsity Normalization performs statistically significantly better than all other baselines with item vector encoding, which is considered a better encoding scheme in most collaborative filtering models (Salakhutdinov et al. 2007 Sedhain et al. 2015 Zheng et al. 2016).
Table 7: Comparison between Sparsity Normalization and other missing handling techniques using AutoRec on Movielens 100K dataset. Test RMSE with 95% confidence interval of 5-runs provided.
Models
User vector
Item vector
Zero Imputation w/o Sparsity Normalization
0.9346 0.0007
0.8835 + 0.0003
Zero Imputation w/ Sparsity Normalization (ours.)
0.9280 + 0.0023
0.8809 + 0.0011
Zero Imputation w/ Batch Normalization
0.9929 + 0.0088
0.9205 = 0.0081
Zero Imputation w/ Layer Normalization
0.9996 0.0131
0.9396 1 0.0141
Dropout
0,9252 1 0.0019
0.9268 1 0.0261
Mean Imputation
0,9310 0.0017
0.9206 + 0.0012
Median Imputation
0.9333 0.0014
0.9196 + 0.0017
k:-N
0.9346 + 0.0007
0.9133 + 0.0011
MICE Buuren Groothuis-Oudshoomm 2010
0,9318 0.0015
0.9209 = 0.0022
SoftImpute
Mazumder ct al. 2010
0.9262 + 0.0003
0.8867 - 0.0007
GMMC
Smicja et al.
2018
0.9331 0.0067
0.9109 + 0.0166
GAIN
Yoon et al.
2018
1.0470 + 0.0098
1.0354 1 0.0101
H.1.2 Is BATCH NORMALIZATION OR LAYER NORMALIZATIONN ABLE TO SOLVE VSP?
Someone might wonder if Batch Normalization (Ioffe & Szegedy 2015 or Layer Normaliza- tion (Lei Ba et al. 2016) have similar effect to Sparsity Normalization by alleviating VSP However, BN or LN can not solve VSP even though these three methods have something in common in terms of stabilizing the statisties of the hidden layer. To validate this, compare SN with LN and BN by controlling the strength of weight decay regularization. We use the AutoRec model with Movielens 100K for these experiments as in Appendix H.I.I
Figure9]showe shows that VSP occurs in all cases except SN with weak regularization (left column). The model's 's prediction highly correlates the number of known entries in all methods except SN. On the other hand, strong regularization might seem to solve the VSP while the model shows relatively constant inference regardless of the number of known entrics. However, the strong regularization is not an acceptable solution because it gives the less freedom of the model's inference making constant prediction (right column). It must not be natural that the predicted values of the model are almost constant regardless of input sample (we do not want a model that recommends the same movie no
Only when applying Batch Normalization and Layer Normalization, we set the number of early stop iteration 10, 000 (10 times of the other models) in order to prevent underfitting.
23

---

Published as a conference paper at ICLR 2020
5=50
A=500
A=5000
3.75
model prediction
3.50
3.25
3.00
2.75
30
zero imputation w/ SN
model prediction
3.0
2.5
2.0
zero imputation w/ LN
2.0
model prediction
1.5
1.0
0.5
zero imputation w/ BN
1.5
model prediction
1.0
30
entries
Figure 9: Predicted values of AutoRec (Sedhain et al. 2015) (user vector encoding) on Movielens 100K (collaborative filtering) dataset with zero imputation (w/ or w/o normalization) according to the number of known entries for randomly selected test point. Input masks are randomly sampled (to artificially control its sparsity level). For each target sparsity level through X-axis, 100 samples are drawn, scattering the predicted values and plotting the average in solid line. Predicted values of the model is also plotted according to the strength the weight decay controlling of the hyper-parameter for weight decay. First row: The vanilla zero imputation. Second row (ours.): Zero imputation with Sparsity Normalization. Third row: Zero imputation with Layer Normalization (Lei Ba et (0066). Fourth row: Zero imputation with Batch Normalization (Toffe Szegedy 2015). First column:
Weak regularization (A 50). Second column: Moderate regularization 500). Third column: Strong regularization (A 5000).
matter which movies user like/dislike!). This trend can also be seen in the process of tuning the hyper-parameter À for each model through that the optimal value of each model except LN and BN determined at around 500, whereas that of BN and LN is determined above 500000 (inordinate regularization). In other words, unlike SN, the VSP is not solved with I.N or BN. Rather, strong regularization able to solve the VSP, but this is not direct solution to VSP forcing the model to choice constant predicted values irrespectively of the input. It is instructive note that the trend of Figure 9 is extremely consistent with the test points Figure] 1
24

---

Published as a conference paper at ICLR 2020
H.2 ELECTRONIC MEDICAL RECORDS (EMR) DATASETS
We also compare Sparsity Normalization and baselines for the five disease identification tasks in the NHIS dataset used in Section 4.2) The results are described in Table Sparsity Normalization shows better or similar performance compared to other baseline methods as well.
H.3 UCI DATASETS
We further compare Sparsity Normalization with other missing handling techniques UCI datasets which have relatively low missing rates and small feature dimension (non-high dimensional datasets). We consider the UCI datasets used in GAIN Yoonn et al. 2018 and GMMC (Smieja et al. 2018 The datasets used in the both papers can be divided into two categories: missing features are intentionally injected (w/ MCAR assumption) or missing features exist inherently (w/o MCAR assumption).
We adopt the settings of Klambauer et al. (2017): Littwin & Wolf (0188) 's UCI exeperiments to ReLU (Glorot et al. 201T) networks of hidden layers with 256 units. The main purpose imputation should be to improve prediction performance rather than imputation performance. In this reason, we just focus on the prediction performance of each missing handling techniques for UCI datasets. Because all UCI datasets used in this section for imbalanced binary classification tasks, prediction performance is reported with AUROC rather than accuracy, and the class weight is considered in loss function. On top of that, we use Adam Optimizer in all experiments for fair comparison with baselines.
use the same Multi Layer Perceptron (MLP) architecture as Section
Though we adopt datasets used in GAIN (oon et al. 2018 and GMMC (Smieja et al. 2018), we report quite different performance from that of the papers. Several possible reasons are as follows. First, GAIN and GMMC did not publish the train/test dataset split, thus we use our own split which made under the similar settings of both papers. Second, MLP used rather than logistic regression or Radial Basis Function Network (RBFN) which are used in GAIN and GMMC respectively. is because think that MIP is more reasonable and widely acceptable architecture nowaday than the others. Furthermore, final possible reason is that we use Adam Optimizer for all models because SGD with learning rate decay is difficult for fair comparison when hyper- hyper- parameters are set in favor to particular model. In these ways, we do our best to compare Sparsity Normalization and other missing handling techniques including GMMC and GAIN in the most fair and reasonable setting.
AUROC and class weights, unlike the GAIN and GMMC. The
Table Summary of UCI datasets with and without MCAR assumption.
w/ MCAR ussumption
w/o MCAR assumpption
Dulaset
Breast
Spam
Credit
Crashes
Heart
Bands
Hepartitis
Horse
Mammographics
Pima
# Instances
569
4601
30000
540
270
539
155
368
961
768
Attributes
30
57
23
20
5
Missing Rate (%)
20.0
20.0
20.0
5.38
3.37
12.2
22 Although there are MLP experiments on GMMC, they cover only one dataset and used too small capacity. also hard get results similar what the author reported in spite of running the author's public code.
25

---

Published as a conference paper at ICLR 2020
H.3.1 UCI DATASETS WITH MCAR ASSUMPTION
We deliberately inject missing values into the datasets which don't have any missing attributes inter- nally (w/ MCAR assumption) to perform binary classification tasks. We consider Breast, Spam, and Credit datasets from GAIN (Yoon al. 2018) and Crashes and Heart datasets from GMMC (Smieja et al. 2018). We make 20% of all features be missing for the Breast, Spam, and Credit datasets following GAIN, and 50% for the Crashes and Heart datasets following GMMC. The summary of the datasets are described in Table For Breast, Spam, and Credit datasets taken by GAIN, min-max normalization which makes min and max values of each features be 0 and 1 following GAIN paper, and for Crashes and Heart datasets taken by GMMC, we perform another kind of min-mat max normalization which makes min and max values of each features be -1 and 1 following GMMC paper.
perform
Table Comparison between Sparsity Normalization and other missing handling techniques on five imbalanced binary classification tasks in UCI datasets with MCAR assumption. Test AUROC with 95% confidence interval of 5-runs is provided.
Models
Breast
Spam
Credit
Crashes
Heart
Zero Imputation w/o SN
0.9958
0.0067
0.9699
0.0040
0.7344
0.0135
0.9070 + 0,1050
0.8967 +: 0.0268
Zero Imputation SN (ours.)
0,9992
0.0023
0,9700 = 0.0032
0.7292 + 0.0098
0.8905 + 0.0934
0.9055 + 0.0327
Zero Imputation
L,0000
0.0000
0.9666
0.0051
0.7178
0.0143
0.9077
0.0645
0,8857 + 0,0268
Zero Imputation LN
0.9979
0.0059
0,9704
0.0031
0.7353
0.0051
0,9095
0.0972
0.8863 = 0.0177
Dropout
0.9897
0.0209
0.9697
0.0031
0.7313
0.0066
0,9220
0.0553
0.8K57 0.0335
Mean Imputation
0.9966
0.0089
0.9690
0.0028
0.7349 + 0.0066
0.8990 + 0.1081
0.9011 + 0.0522
Median Imputation
0.9987 + 0.0045
0.9695
0.0024
0.7353 1 0.0044
0.9080 : 0.0892
0.9253 : 0.0407
k-NN
1.0000 + 0.0000
0.9613 = 0.0035
0,5270 + 0.0171
0.8940 + 0.0356
0.8703 + 0.0435
MICE
1.0000 0.0000
0,9680 = 0.0041
0,5315 + 0.0187
0.9760 : 0.0424
0.9066 + 0.0410
SoftImpute
1,0000 0.0000
0.9725
0.0021
0,5375 + 0.0184
0.9390 + 0.0754
0,8846 + 0.0437
GMMC
0.9987 0.0026
0.9679
0.0029
0.7355
0.0053
0,9400 = 0,0422
0,9264 = 0.0223
GAIN
1.MM00 0.0000
0.9688 0.0029
0.7356 0.0045
0.9650 0.0360
0.8879 0.0196
The experimental results are summarized in Table It is difficult to find significant difference in prediction performance among each missing handling techniques for datasets with small feature dimensions. The results of these experiments are also consistent with experiments of GAIN (Yoon eet al. 2018). Though the GAIN showed significantly better imputation performance compared to their baseline methods, prediction performances were not statistically significant (See Table 3 of the GAIN paper, and note that they didn't 't report 95% confidence interval but standard deviation). From these overall results, we conclude that SN is quite comparable for the datasets of low dimension/missing rate with MCAR assumption.
26

---

Published as a conference paper at ICLR 2020
H.3.2 UCI DATASETS WITHOUT MCAR ASSUMPTION
We compare SN and each missing handling techniques on the datasets that have internal missingness (wlo MCAR assumption). We consider Bands, Hepartítis, Horse, Mammographics, and Pima datasets experimented in the GMMC (Smieja et al. 2018) paper (See Table for statistics of the datasets). Following the GMMC paper, the min-max normalization performed, which makes min and max values of each features be -1 and 1. As shown in Table [0] it is concluded that even without MCAR assumption, SN shows comparable results for the datasets of low dimension/missing rate.
Table 10: Comparison between Sparsity Normalization and other missing handling techniques on five imbalanced binary classification tasks in UCI datasets without MCAR assumption. Test AUROC with 95% confidence interval of 5-runs provided.
Models
Bands
Hepartitis
Horse
Mammographics
Pima
Zero Imputation w/o SN
0.7851 + 0.0962
0,8222 = 0.0826
0.9090 + 0.0331
0,8835 + 0,0083
0.8466 = 0,0180
Zero Imputation w/ SN (ours.)
0.7939 + 0.1011
0.8556 = 0.1300
0.9238 + 0.0311
0,8787 1 0.0063
0.8355 + 0.0233
Zero Imputation w/ BN
0.7512 = 0.0716
0.7889 = 0.1061
0.8827 + 0.0362
0.8933 + 0.0055
0.8614 + 0.0106
Zem Imputation w/ LN
0.8000 0.0789
0.8222 0.0911
0.9115 0.0404
0.8854 1 0.0097
0.7972 : 0.0074
Dropout
0,7956 0.1012
0.8194 = 0.1089
0.9034 + 0.0507
0.8816 + 0.0059
0.8408 = 0.0292
Mean Imputation
0.7765 0.0504
0,8556
0.0895
0.9009 0.0271
0.8834 : 0.0113
0.8382
0,0142
Median Imputation
0.7997 0.0682
0,8889
0.0544
0.9133 0.0379
0.8783 + 0.0070
0.8334 0.0099
k-NN
0.7571 + 0.0430
0,8833
0.1047
0.9300 + 0.0363
0,8840 + 0.0141
0.6237 = 0.0399
MICE
0.7618 0.0297
0,8133
0.06667
0.9195 + 0.0431
0,8814 + 0.IM877
0.6477 + 0.0655
SoftImpute
0.7644 + 0.0475
0,8139 = 0.1047
0.9232 + 0.0124
0.8848 + 0.0089
0,59 + 0.0341
GMMC
0.7985 0.0605
0.8500
0.0621
0.9000 1 0.0124
0.8844 : 0.0090
0.8391
1 0.0201
GAIN
0.0785
0.0826
0.8941
0.0322
0.8859
0.0099
0,8457
0.0178
H.4 CONCLUSION
In conclusion, Sparsity Normalization is significantly superior to other missing handling techniques for the high dimensional/missing rate datasets. Sparsity Normalization performs well compared other modern missing handling techniques even on non-high dimensional datasets. Sparsity Normalization is valuable in that it performs better than other models and does not require additional training parameters. Moreover, Sparsity Normalization is computationally inexpensive compared to Mean or Median Imputation because sparse tensors can be used to save computational costs to calculate first hidden layer not with mean or median imputation but with zero imputation (w/ or w/o SN). The reduced computational cost by using sparse tensors is relatively higher when we deal with high-dimensional datasets.
27


Reviews:
[{"review_id": "rkgNqJ7iKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a novel solution to the variable sparsity problem, where the output of neural networks biased with respect to the number of missing inputs. The authors proposed a sparsity normalization algorithm to process the input vectors to encounter the bias. In experiments, the authors evaluated the proposed sparsity normalization model on multiple datasets: collaborative filtering datasets, electric medical records datasets, single-cell RNA sequence datasets and UCI datasets. Results show that the proposed normalization method improves the prediction performance and the predicted values of the neural network is more uniformly distributed according to the number of missing entries.\n\nThe paper describes a clear and specific machine learning problem. Then the authors demonstrate a simple normalization strategy is capable of fixing the issue of biased prediction. The paper has a well-organized structure to convey the motivation. Therefore, my opinion on this paper leans to an acceptation. My questions are mainly on the experiment section:\n\n1) As shown in Table 2, there are various new collaborative filtering methods proposed after 2015, why the authors chose to extend AutoRec (Sedhain et al., 2015) but not other new methods?\n\n2) In the experiments, you compare your model with zero imputation (Please correct me if w/o SN is not zero imputation). However, I think it is a common practice in machine learning that we perform imputation with mean or median values. I'm interested in knowing whether filling with mean/median values work with these datasets.\n\n3) In section 4.5, you mentioned that \"SN is effective\neven when MCAR assumption is not established\". However, I'm still not clear about the reason. I believe many machine learning datasets have NMAR (not missing at random) type of missing data, but not MCAR. So this is an important issue for me.\n\n4) Does your model assume all input values are numerical but not categorical? ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "r1lmuyH9tr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Zero imputation is studied from a different view by investigating its impact on prediction variability and a normalization scheme is proposed to alleviate this variation. This normalization scales the input neural network so that the output would not be affected much. While such simple yet helpful algorithms are plausible there are number of remaining issues:\n1-\tZero imputation, as authors mentioned, is not an acceptable algorithm for imputation and improving on that via the normalization proposed in the paper cannot be counted as an exciting move in this area unless an extensive comparison shows it’s benefits over the many other existing techniques. I am interested to see how would the results be if you compare this simple algorithm with more complicated ones like GAIN or MisGAN. It is argued in the paper that with high dimensional data, your algorithm is more acceptable, but how would it be with in other cases?\n2-\tYour algorithm is only explained with neural net framework, how can we extend it to the other machine learning models?\n3-\tIs batch normalization used in your experiments? Scaling the activation in one layer to reduce its impact on the next layer is somehow similar to what happens in batch normalization, and I am wondering if BN makes any similar effect?\n4-\tPlease provide labels for the x-axes in the figures.\n\n------------------------------------------\nAfter rebuttal:\nThanks for adding the extra experiments.\nLooking at Table 9 in appendix, I am bit surprised to see that sometimes mean imputation works better than MICE (GAIN usually works good with large data). Maybe it attributes to the missing features. How did you choose to apply 20% missingness? randomly?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1eWppVRFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies a very interesting phenomena in machine learning called VSP, that is the output of the model is highly affected via the level of missing values in its input.  The authors demonstrate the existence of such phenomena empirically, analyze the root cause for it theoretically, and propose a simple yet effective normalization method to tackle the problem. Several experiments demonstrate the effectiveness of this method.\n\nIn general I think the paper is descent and elegant. It is motivated from real-world pain-point, gives a rigorous study towards the root cause, and the proposed method is very effective. To the best of my knowledge there is no prior work looking deep into this area and this paper does bring new insights to the community. As a result I would vote for its acceptance.\n\nOne issue is that I find the backbone methods in experiments are somehow out-of-date. For example, AutoRec (2015) and CF-NADE (2016). I admit that I’m not an expert in the field of recommendation but still think that more recent, and powerful baseline algorithms should be applied on to further demonstrate the true effectiveness of Sparsity Normalization.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the problem of using zero imputation when input features are missing. The authors study this problem, propose a solution, and evaluate on several benchmark datasets. The reviewers were generally positive about the paper, but had some questions and concerns about the experimental results. The authors addressed these concerns in the rebuttal. The reviewers are generally satisfied and believe that the paper should be accepted.
Decision: Accept (Poster)