Why Not to Use Zero Imputation? Correcting Sparsity Bias in Training Neural Networks

Handling missing data is one of the most fundamental problems in machine learning . Among many approaches , the simplest and most intuitive way is zero imputation , which treats the value of a missing entry simply as zero . However , many studies have experimentally confirmed that zero imputation results in suboptimal performances in training neural networks . Yet , none of the existing work has explained what brings such performance degradations . In this paper , we introduce the variable sparsity problem ( VSP ) , which describes a phenomenon where the output of a predictive model largely varies with respect to the rate of missingness in the given input , and show that it adversarially affects the model performance . We first theoretically analyze this phenomenon and propose a simple yet effective technique to handle missingness , which we refer to as Sparsity Normalization ( SN ) , that directly targets and resolves the VSP . We further experimentally validate SN on diverse benchmark datasets , to show that debiasing the effect of input-level sparsity improves the performance and stabilizes the training of neural networks . 1 INTRODUCTION . Many real-world datasets often contain data instances whose subset of input features is missing . While various imputing techniques , from imputing using global statistics such as mean , to individually imputing by learning auxiliary models such as GAN , can be applied with their own pros and cons , the most simple and natural way to do this is zero imputation , where we simply treat a missing feature as zero . In neural networks , at first glance , zero imputation can be thought of as a reasonable solution since it simply drops missing input nodes by preventing the weights associated with them from being updated . Some what surprisingly , however , many previous studies have reported that this intuitive approach has an adverse effect on model performances ( Hazan et al. , 2015 ; Luo et al. , 2018 ; Śmieja et al. , 2018 ) , and none of them has investigated the reasons of such performance degradations . In this work , we find that zero imputation causes the output of a neural network to largely vary with respect to the number of missing entries in the input . We name this phenomenon Variable Sparsity Problem ( VSP ) , which should be avoided in many real-world tasks . Consider a movie recommender system , for instance . It is not desirable that users get different average of predicted ratings just because they have rated different number of movies ( regardless of their actual rating values ) . One might argue that people with less ratings do not like movies in general and it is natural to give higher predicted values to people with more ratings . This might be partially true for users of some sparsity levels , but it is not a common case uniformly applicable for a wider range of sparsity levels . This can be verified in real collaborative filtering datasets as shown in Figure 1 ( upper left corner ) where users have a similar average rating for test data regardless of the number of known ratings ( see also other two examples in Figure 1 ) . However , in standard neural networks with zero imputation , we observe that the model ’ s inference correlates with the number of known entries of the data instance as shown in the second row of Figure 11 . It would be fatal in some safety-critical applications such as a medical domain : a patient ’ s probability of developing disease for example should not be evaluated differently depending on the number of medical tests they received ( we do not want our model to predict the probability of death is high just because some patient has been screened a lot ! ) . In addition , we theoretically analyze the existence of VSP under several circumstances and propose a simple yet effective means to suppress VSP while retaining the intuitive advantages of zero imputation : normalizing with the number of non-zero entries for each data instance . We refer to this regularization as Sparsity Normalization , and show that it effectively deals away with the VSP , resulting in significant improvements in both the performance and the stability of training neural networks . Our contribution in this paper is threefold : • To best of our knowledge , we are the first in exploring the adverse effect of zero imputation , both theoretically and empirically . 1Note that , this tendency is very consistent with other test points and is observed throughout the entire learning process ( even before the training ) . • We identify the cause of adverse effect of zero imputation , which we refer to as variable sparsity problem , and formally describe how this problem actually affects training and inference of neural networks ( Section 2 ) . We further provide new perspectives using VSP to understand phenomena that have not been clearly explained or that we have misunderstood ( Section 4 and 5 ) . • We present Sparsity Normalization ( SN ) and theoretically show that SN can solve the VSP under certain conditions ( Section 3 ) . We also experimentally reaffirm that simply applying SN can effectively alleviate or solve the VSP yielding significant performance gains ( Section 4 ) . 2 VARIABLE SPARSITY PROBLEM . We formally define the Variable Sparsity Problem ( VSP ) as follows : a phenomenon in which the expected value of the output layer of a neural network ( over the weight and input distributions ) depends on the sparsity ( the number of zero values ) of the input data ( Figure 2a ) . With VSP , the activation values of neural networks could become largely different for exactly the same input instance , depending on the number of zero entries ; this makes training more difficult and may mislead the model into incorrect predictions . While zero imputation is intuitive in the sense that it drops the missing input features , we will show that it causes variable sparsity problem for several example cases . Specifically , we show the VSP under assumptions with increasing generality : ( Case 1 ) where activation function is an identity mapping with no bias , ( Case 2 ) where activation function is an affine function , and ( Case 3 ) where activation function is a non-decreasing convex function such as ReLU ( Glorot et al. , 2011 ) , leaky ReLU ( Maas et al. , 2013 ) , ELU ( Clevert et al. , 2016 ) , or Softplus ( Dugas et al. , 2001 ) . Here , we summarize the notation for clarity . For a L-layer deep network with non-linearity σ , we use W i ∈ Rni×ni−1 to denote the weight matrix of i-th layer , bi ∈ Rni to denote the bias , hi ∈ Rni to denote the activation vector . For simplicity , we use h0 ∈ Rn0 and hL ∈ RnL to denote input and output layer , respectively . Then , we have hi = σ ( W ihi−1 + bi ) , for i = 1 , · · · , L. Our goal in this section is to observe the change in hL as the sparsity of h0 ( input x ) changes . To simplify the discussion , we consider the following assumption : Assumption 1 . ( i ) Every coordinate of input vector , h0l , is generated by the element-wise multiplication of two random variables h̃0l and ml where ml is binary mask indicating missing value and h̃0l is a ( possibly unobserved ) feature value . Here , missing mask ml is MCAR ( missing completely at random ) , with no dependency with other mask variables or their values h̃0 . All ml follow some identical distribution with mean µm . ( ii ) The elements of matrix W i are mutually independent and follow the identical distribution with mean µiw . Similarly , b i and h̃0 consist of i.i.d . coordinates with mean µib and µx , respectively . ( iii ) µ i w is not zero uniformly over all i . ( i ) assumes the simplest missing mechanism . ( ii ) is similarly defined in Glorot & Bengio ( 2010 ) and He et al . ( 2015 ) in studying weight initialization techniques . ( iii ) may not hold under some initialization strategies , but as the learning progresses , it is very likely to hold . ( Case 1 ) For simplicity , let us first consider networks without the non-linearity nor the bias term . Theorem 1 shows that the average value of the output layer E [ hLl ] is directly proportional to the expectation of the mask vector µm : Theorem 1 . Suppose that activation σ is an identity function and that bil is uniformly fixed as zero under Assumption 1 . Then , we have E [ hLl ] = ∏L i=1 ni−1µ i wµxµm . ( Case 2 ) When the activation function is affine but now with a possibly nonzero bias , E [ hLl ] is influenced by µm in the following way : Theorem 2 . Suppose that activation σ is an affine function under Assumption 1 . Suppose further that fi ( x ) is defined as σ ( ni−1µiwx+ µ i b ) . Then , E [ h L l ] = fL ◦ · · · ◦ f1 ( µxµm ) . ( Case 3 ) Finally , when the activation function is non-linear but non-decreasing and convex , we can show that E [ hLl ] is lower-bounded by some quantity involving µm : Theorem 3 . Suppose that σ is a non-decreasing convex function under Assumption 1 . Suppose further that fi ( x ) is defined as σ ( ni−1µiwx+ µ i b ) and µ i w > 0 . Then , E [ h L l ] ≥ fL ◦ · · · ◦ f1 ( µxµm ) . If the expected value of the output layer ( or the lower bound of it ) depends on the level of sparsity/missingness as in Theorem 1-3 , even similar data instances may have different output values depending on their sparsity levels , which would hinder fair and correct inference of the model . As shown in Figure 1 ( second row ) , the VSP can easily occur even in practical settings of training neural networks where the above conditions do not hold . 3 SPARSITY NORMALIZATION . Algorithm 1 Sparsity Normalization ( SN ) Input : Dataset D , constant K. Output : Sparsity Normalized Dataset DSN . Empty set S = φ for each ( h0 , m ) ∈ D do h0SN ← K · h0/ ‖m‖1 S ← S ∪ { h0SN } end for DSN ← S In this section , we propose a simple yet surprisingly effective method to resolve the VSP . We first revisit ( Case 2 ) to find a way of making expected output independent of input sparsity level since the linearity in activation simplifies the correction . Recalling the notation of h0 = h̃0 m ( represents the element-wise product ) , we find that simply normalizing via h0SN = ( h̃ 0 m ) ·K1/µm for any fixed constant K1 , can debias the dependency on the input sparsity level . We name this simple normalizing technique Sparsity Normalization ( SN ) and describe it in Algorithm 1 . Conceptually , this method scales the size of each input value according to its sparsity level so that the change in output size is less sensitive to the sparsity level ( Figure 2b ) . The formal description on correcting sparsity bias by SN is as follows in this particular case : Theorem 4 . ( With Sparsity Normalization ) Suppose that activation σ is an affine function under Assumption 1 . Suppose further that fi ( x ) = σ ( ni−1µiwx+ µ i b ) and replace the input layer using SN , i.e . h0SN = ( h̃ 0 m ) ·K1/µm for any fixed constantK1 . Then , we haveE [ hLl ] = fL◦· · ·◦f1 ( µx·K1 ) . Unlike in Theorem 2 , SN in Theorem 4 makes average activation to be independent of µm , which determines the sparsity levels of input . It is not trivial to show the counterpart of ( Case 3 ) using SN since E [ σ ( x ) ] = σ ( E [ x ] ) does not hold in general . However , we show through extensive experiments in Section 4 that SN is practically effective even in more general cases . While Theorem 4 assumes that µm is known and fixed across all data instances , we relax this assumption in practice and consider varying µm across data instances . Specifically , by a maximum likelihood principle , we can estimate µm for each instance by ∥∥h0∥∥ 0 /n0 = ‖m‖1/n0 . Thus , we have h0SN = K · h0/ ‖m‖1 where K = n0 · K1 ( see Algorithm 1 ) 2 . In practice , we recommend using K as the average of ‖m‖1 over all instances in the training set . We could encounter the dying ReLU phenomenon ( He et al. , 2015 ) if K is too small ( e.g. , K = 1 ) . Since the hyper-parameter K can bring in a regularization effect via controlling the magnitude of gradient ( Salimans & Kingma , 2016 ) , we define K = E ( h0 , m ) ∈D [ ‖m‖1 ] so that the average scales remain constant before and after the normalization , minimizing such side effects caused by SN .

Summary: This paper studies a very interesting phenomena in machine learning called VSP, that is the output of the model is highly affected via the level of missing values in its input.  The authors demonstrate the existence of such phenomena empirically, analyze the root cause for it theoretically, and propose a simple yet effective normalization method to tackle the problem. Several experiments demonstrate the effectiveness of this method.