PROMISSING: Pruning Missing Values in Neural Networks

1 INTRODUCTION . Missing and incomplete data are abundant in real-world problems ; however , the learning and inference procedures in machine learning ( ML ) models highly rely on high-quality and complete data . Therefore , it is necessary to develop new methods to deal with data imperfections in rugged environments . Currently , the most popular way to deal with imperfect data is to impute the missing values . However , if we consider the learning and inference procedures in our brain as a role model for ML algorithms , data imputation barely follows the natural principles of incomplete data processing in our brain . This is because the imputation is generally based on using a heuristic for replacing missing values . Our brain does not impute incomplete sensory information but instead uses its incompleteness as a separate source of information for decision making . For example , by only hearing the rain we can estimate how hard it is raining , and we do not necessarily need to receive visual information . Instead , we direct our attention more toward our auditory inputs to decide whether to go out with an umbrella . In addition , the more we miss sensory information , the more cautious we get in decision-making . That is why we are more careful in darker environments . Neural networks ( NNs ) are brain-inspired algorithms that are very popular these days ( under the name of deep learning ) for learning complex relationships between inputs and target variables . However , they are in principle unable to handle incomplete data with missing values . They mainly rely on matrix operations which can not operate on not-a-number ( NaN ) values . Only one NaN in a dataset impairs the forward propagation in a network . There are three solutions to this problem ( Garcı́aLaencina et al. , 2010 ) : i ) removing samples or features with missing values , ii ) imputing the missing values , and iii ) modeling the incomplete data . Removing the samples with missing values can be very costly , especially in small-sample size and high-dimensional datasets . For example , data collection in clinical applications is an expensive procedure in terms of time , finance , and patient burden . Moreover , removing even a few samples from small datasets can affect negatively the generalization performance of the final model . Removing informative features with missing values is also compensated with lower model performance . Therefore , filling the information gaps is inevitable . There are various techniques for data imputation , ranging from simply imputing missing values with a constant to more sophisticated ML-based imputation approaches ( Little & Rubin , 2019 ; Garcı́aLaencina et al. , 2010 ) . One can categorize the most common techniques into three main categories : i ) constant imputation , ii ) regression-based imputation , and iii ) ML-based imputation . In constant imputation , the missing values are replaced with a constant , e.g. , zeros or mean/median of features . It has been shown that constant imputation is Bayes consistent when the missing features are not informative ( Josse et al. , 2019 ) . In the regression-based imputation , a linear or non-linear regression model is derived to predict the missing values . This method can be used to impute a single or multiple features . The most popular regression-based imputation is Multiple Imputation by Chained Equations ( MICE ) ( Van Buuren & Groothuis-Oudshoorn , 2011 ; Azur et al. , 2011 ) . In MICE , an iterative procedure of predicting missing values and re-training regressors with updated predictions is performed for a limited number of cycles . The central assumption behind the MICE approach is that the missing values are missed at random ( see Rubin ( 1976 ) and Appendix A.1 for definitions of missing value mechanisms including missing completely at random ( MCAR ) , missing at random ( MAR ) , and missing not at random ( MNAR ) ) . Applying MICE can result in biased estimations if this assumption is not satisfied ( Azur et al. , 2011 ) . Another critical limitation of regression-based imputation is its high computational complexity ( Caiafa et al. , 2021 ) . If we do not know which feature will be missed at the test time , for d features , we need to train d different regression models . In the ML-based approach ML algorithms , such as a K-nearest neighbor ( KNN ) , regularized linear model ( Jiang et al. , 2021 ) , decision trees ( Twala et al. , 2008 ) , random forest ( Xia et al. , 2017 ) , neural network ( Bengio & Gingras , 1996 ) , or generative model ( Yoon et al. , 2018 ; Ipsen et al. , 2020 ; Collier et al. , 2020 ; Nazabal et al. , 2020 ) , are used for handling missing data . As an alternative solution to data imputation , one can use the elegance of probabilistic modeling to model the incomplete data under certain assumptions . One seminal work in this direction is presented by Ghahramani & Jordan ( 1994 ) , where a Gaussian Mixture Model ( GMM ) is used to estimate the joint density function on incomplete data using an Expectation-Maximization ( EM ) algorithm . This approach is later adopted and extended to logistic regression ( Williams et al. , 2005 ) , Gaussian processes , support vector machines ( Smola et al. , 2005 ) , and multi-class non-linear classification ( Liao et al. , 2007 ) . However , despite their good performance on small-size datasets , their application remained limited on big and high-dimensional data due to the high computational complexity ( Caiafa et al. , 2021 ) . To overcome this issue , Caiafa et al . ( 2021 ) proposed a sparse dictionary learning algorithm that is trained end-to-end , and simultaneously learns the parameters of the classifier and sparse dictionary representation . Le Morvan et al . ( 2020 ) proposed NeuMiss , a neural-network architecture that uses a differentiable imputation procedure in a impute-then-regress scheme ( Morvan et al. , 2021 ) . A notable feature of NeuMiss is its robustness to MNAR data . Inverse probability weighted estimation ( Wooldridge , 2007 ; Seaman & White , 2013 ) is another probabilistic approach for handling missing values without imputation in which the weights for samples with many missing values are inflated based on an estimation of the sampling probability . Recently , Smieja et al . ( 2018 ) proposed a modified neuron structure that uses GMM with a diagonal covariance matrix ( assuming MAR ) to estimate the density of missing data . GMM parameters are learned with other network parameters . Conveniently , it handles missing values in the first layer of the network , and the rest of the architecture remains unchanged . Elsewhere , Nowicki et al . ( 2016 ) proposed a new neural network architecture based on rough set theory ( Pawlak , 1998 ) for learning from imperfect data . It is fascinating that this method can say “ I do not know ” when a large portion of input values are missing , unlike traditional models trained on imputed data that may predict definite outcomes even on completely unmeasured samples , i.e. , they run in the absolute darkness . These predictions can be dangerous with catastrophic consequences in more delicate applications of ML for example in autonomous driving , robotic surgery , or clinical decision-making . In this work , we attack the problem of modeling incomplete data using artificial neural networks without data imputation . We propose a simple technique for pruning missing values ( PROMISSING ) in which the effect of missing values on the activation of a neuron is neutralized . In this strategy , a missing value is not replaced by arbitrary values ( e.g. , through imputation ) ; it is naturally considered a missing piece of the puzzle ; we learn a problem-specific numerical representation for unknowns . The key feature of PROMISSING is its simplicity ; it is plug-and-play ; it deals with missing values in the first layer of the network without the need to change anything in the rest of the network architecture or optimization process . PROMISSING in its original form does not add extra parameters to the network , and its computational overhead remains negligible . Our experiments on simulated data and several classification/regression problems show that the proposed pruning method does not negatively affect the model accuracy and provides competitive results compared to several data imputation techniques . In a clinical application , making prognostic predictions for patients with a psychotic disorder , we present an application of PROMISSING on a multi-modal clinical dataset . We demonstrate how the NN model trained using PROMISSING becomes indecisive when facing many unknowns . This is a crucial feature for developing trustworthy prediction models in clinical applications . Furthermore , we show a side application of PROMISSING for counterfactual interpretation ( Mothilal et al. , 2020 ) of NNs decisions that can be valuable in clinics . 2 METHODS . Let x ∈ Rp represent a vector of an input sample with p features . We assume that the features in x are divided into two sets of q observed xo ∈ Rq and r missing features xm ( where p = q + r ) . In this study , we do not put any assumption on the pattern of missing values in x . Then , the activation of the kth ( k ∈ { 1 , 2 , . . . , s } ) neuron in the first hidden layer of an ordinary NN is : a ( k ) = ∑ xi∈xo xiw ( k ) i + ∑ xj∈xm xjw ( k ) j + b ( k ) . ( 1 ) This activation can not be computed unless the values in xm are imputed with real numbers . Here , in PROMISSING , we propose to alternatively replace the missing values with a neutralizer that 1 ) prunes the missing values from inputs of a neuron , 2 ) neutralizes the effect of missing values on the neuron ’ s activation by cancelling the second term in Eq . 1 and modifying the neuron ’ s bias . A missing value xj ∈ xm is replaced with its corresponding neutralizer u ( k ) j at the kth neuron , where : u ( k ) j = −b ( k ) pw ( k ) j . ( 2 ) The value of a neutralizer depends on its corresponding weight ( w ( k ) j ) , the bias of the corresponding neuron ( b ( k ) ) , and the number of features ( p ) ; thus , it can be computed on the fly during the training or inference procedures . A small value is added to weights before computing neutralizers to avoid division by zero . Inserting the neutralizer into Eq . 1 , the activation of the kth neuron is rewritten as : a ( k ) = ∑ xi∈xo xiw ( k ) i + qb ( k ) p , ( 3 ) in which the effect of weights of missing values on the activation of the neuron is eliminated , and the neuron ’ s bias is reduced by a factor of r/p . If all input values for a specific sample are missing then the neuron is completely neutralized . Proposition 1 If all input values are missing ( q = 0 and xo = ∅ ) then the activation of a PROMISSING neuron is zero ( see Appendix A.2.2 for the proof ) . Proposition 2 If there are no missing values in inputs ( q = p and xm = ∅ ) then the activation of a PROMISSING neuron is equal to a normal neuron ( see Appendix A.2.3 for the proof ) .. We should emphasize that , when using PROMISSING , the user does not need to apply any change to the input vectors , and the missing values ( generally represented as nans in the input matrix ) are fed directly to the network . After the training procedure , we eventually learn U ∈ Rs×p , a matrix representation for unknowns ( or metaphorically the dark matter ) : u ∗ ( k ) j = −b∗ ( k ) pw ∗ ( k ) j j ∈ { 1 , 2 , . . . , p } , k ∈ { 1 , 2 , . . . , s } , ( 4 ) where b∗ and w∗ are representing the final learned bias and weight . At the prediction stage , a missing value at jth input feature will be replaced with its corresponding neutralizer from U . It is worth emphasizing that the missing values are replaced with different neutralizers at different neurons ; therefore , it can not be considered a constant imputation . In fact , each neuron perceives differently a missing value in the input space . Metaphorically , the neurons can be seen as blind men in the parable of “ the blind men and an elephant ” 1 when facing unknowns . Furthermore , it is different from regression-based imputation and model-based approaches in the sense that a missing value in a specific feature is not inferred from other observed features , or the distribution of observed values ; i.e. , unknowns remain unknowns . In PROMISSING , we do not assume any certain missing value mechanism ( e.g. , MAR ) in advance . Instead , we try to learn the patterns of missing values from data that maybe advantageous in more difficult scenarios such as MNAR ( see results in Sec . 3.1 ) . One possible drawback of using PROMISSING is in high-dimensional input spaces and when the number of missing values is large , i.e. , when p → ∞ and r q . In this case , the neuron will undershoot ; hence the effect of few non-missing values are ignored . 2 To address this problem , we propose a modified version of PROMISSING ( mPROMISSING ) in which the effect of large r can be compensated with a compensatory weight , wc . The compensatory weight receives a fixed input of r/p for a specific sample ; thus , the activation of the neuron will change to : a ( k ) = ∑ xi∈xo xiw ( k ) i + qb ( k ) + rw ( k ) c p . ( 5 ) w ( k ) c is learned alongside the rest of the network parameters in the optimization process . On training data with few missing values , data augmentation ( e.g. , by simulating different patterns and size of missing values ) is advisable to ensure the sensibility of the learned compensatory weight . Since the input for this weight ( r/p ) is computed at the run time , no modification to input vectors is required . Proposition 3 If there are no missing values ( q = p , r = 0 , and xm = ∅ ) then the activation of an mPROMISSING neuron is equal to a normal neuron ( see Appendix A.2.4 for the proof ) . The proposed PROMISSING approach is straightforward to implement and use . It can be incorporated into the current implementations of different types of NN layers by adding/modifying few lines of code . We have implemented a nanDense layer , inheriting from Keras ( Chollet et al. , 2015 ) Dense layer , using PROMISSING and mPROMISSING neurons ( see appendix A.3 ) . The nanDense layer can be directly imported and used with any Keras model . In its general usage , the nanDense layer is only used for the first layer of an NN to handle missing values in inputs , unless we expect some missing values in the intermediate layers .

Summary: The present paper proposes an alternative to imputation or list-wise deletion in the context of neural networks and incomplete features. Missing values are replaced by a data-specific numerical representation that is learned at the same time as the rest of the network. The handling of the missing values is located in the neurons of the first layer and each missing value is "replaced" by a neuron-specific neutralizer in its activation function. They empirically show that their approach is comparable to several imputation techniques. In a clinical application they show that their model becomes indecisive as the amount of missingness (in terms of missing features) increases, which is a potentially interesting feature for clinical prediction models.