PROMISSING: Pruning Missing Values in Neural Networks

1 INTRODUCTION . Missing and incomplete data are abundant in real-world problems ; however , the learning and inference procedures in machine learning ( ML ) models highly rely on high-quality and complete data . Therefore , it is necessary to develop new methods to deal with data imperfections in rugged environments . Currently , the most popular way to deal with imperfect data is to impute the missing values . However , if we consider the learning and inference procedures in our brain as a role model for ML algorithms , data imputation barely follows the natural principles of incomplete data processing in our brain . This is because the imputation is generally based on using a heuristic for replacing missing values . Our brain does not impute incomplete sensory information but instead uses its incompleteness as a separate source of information for decision making . For example , by only hearing the rain we can estimate how hard it is raining , and we do not necessarily need to receive visual information . Instead , we direct our attention more toward our auditory inputs to decide whether to go out with an umbrella . In addition , the more we miss sensory information , the more cautious we get in decision-making . That is why we are more careful in darker environments . Neural networks ( NNs ) are brain-inspired algorithms that are very popular these days ( under the name of deep learning ) for learning complex relationships between inputs and target variables . However , they are in principle unable to handle incomplete data with missing values . They mainly rely on matrix operations which can not operate on not-a-number ( NaN ) values . Only one NaN in a dataset impairs the forward propagation in a network . There are three solutions to this problem ( Garcı́aLaencina et al. , 2010 ) : i ) removing samples or features with missing values , ii ) imputing the missing values , and iii ) modeling the incomplete data . Removing the samples with missing values can be very costly , especially in small-sample size and high-dimensional datasets . For example , data collection in clinical applications is an expensive procedure in terms of time , finance , and patient burden . Moreover , removing even a few samples from small datasets can affect negatively the generalization performance of the final model . Removing informative features with missing values is also compensated with lower model performance . Therefore , filling the information gaps is inevitable . There are various techniques for data imputation , ranging from simply imputing missing values with a constant to more sophisticated ML-based imputation approaches ( Little & Rubin , 2019 ; Garcı́aLaencina et al. , 2010 ) . One can categorize the most common techniques into three main categories : i ) constant imputation , ii ) regression-based imputation , and iii ) ML-based imputation . In constant imputation , the missing values are replaced with a constant , e.g. , zeros or mean/median of features . It has been shown that constant imputation is Bayes consistent when the missing features are not informative ( Josse et al. , 2019 ) . In the regression-based imputation , a linear or non-linear regression model is derived to predict the missing values . This method can be used to impute a single or multiple features . The most popular regression-based imputation is Multiple Imputation by Chained Equations ( MICE ) ( Van Buuren & Groothuis-Oudshoorn , 2011 ; Azur et al. , 2011 ) . In MICE , an iterative procedure of predicting missing values and re-training regressors with updated predictions is performed for a limited number of cycles . The central assumption behind the MICE approach is that the missing values are missed at random ( see Rubin ( 1976 ) and Appendix A.1 for definitions of missing value mechanisms including missing completely at random ( MCAR ) , missing at random ( MAR ) , and missing not at random ( MNAR ) ) . Applying MICE can result in biased estimations if this assumption is not satisfied ( Azur et al. , 2011 ) . Another critical limitation of regression-based imputation is its high computational complexity ( Caiafa et al. , 2021 ) . If we do not know which feature will be missed at the test time , for d features , we need to train d different regression models . In the ML-based approach ML algorithms , such as a K-nearest neighbor ( KNN ) , regularized linear model ( Jiang et al. , 2021 ) , decision trees ( Twala et al. , 2008 ) , random forest ( Xia et al. , 2017 ) , neural network ( Bengio & Gingras , 1996 ) , or generative model ( Yoon et al. , 2018 ; Ipsen et al. , 2020 ; Collier et al. , 2020 ; Nazabal et al. , 2020 ) , are used for handling missing data . As an alternative solution to data imputation , one can use the elegance of probabilistic modeling to model the incomplete data under certain assumptions . One seminal work in this direction is presented by Ghahramani & Jordan ( 1994 ) , where a Gaussian Mixture Model ( GMM ) is used to estimate the joint density function on incomplete data using an Expectation-Maximization ( EM ) algorithm . This approach is later adopted and extended to logistic regression ( Williams et al. , 2005 ) , Gaussian processes , support vector machines ( Smola et al. , 2005 ) , and multi-class non-linear classification ( Liao et al. , 2007 ) . However , despite their good performance on small-size datasets , their application remained limited on big and high-dimensional data due to the high computational complexity ( Caiafa et al. , 2021 ) . To overcome this issue , Caiafa et al . ( 2021 ) proposed a sparse dictionary learning algorithm that is trained end-to-end , and simultaneously learns the parameters of the classifier and sparse dictionary representation . Le Morvan et al . ( 2020 ) proposed NeuMiss , a neural-network architecture that uses a differentiable imputation procedure in a impute-then-regress scheme ( Morvan et al. , 2021 ) . A notable feature of NeuMiss is its robustness to MNAR data . Inverse probability weighted estimation ( Wooldridge , 2007 ; Seaman & White , 2013 ) is another probabilistic approach for handling missing values without imputation in which the weights for samples with many missing values are inflated based on an estimation of the sampling probability . Recently , Smieja et al . ( 2018 ) proposed a modified neuron structure that uses GMM with a diagonal covariance matrix ( assuming MAR ) to estimate the density of missing data . GMM parameters are learned with other network parameters . Conveniently , it handles missing values in the first layer of the network , and the rest of the architecture remains unchanged . Elsewhere , Nowicki et al . ( 2016 ) proposed a new neural network architecture based on rough set theory ( Pawlak , 1998 ) for learning from imperfect data . It is fascinating that this method can say “ I do not know ” when a large portion of input values are missing , unlike traditional models trained on imputed data that may predict definite outcomes even on completely unmeasured samples , i.e. , they run in the absolute darkness . These predictions can be dangerous with catastrophic consequences in more delicate applications of ML for example in autonomous driving , robotic surgery , or clinical decision-making . In this work , we attack the problem of modeling incomplete data using artificial neural networks without data imputation . We propose a simple technique for pruning missing values ( PROMISSING ) in which the effect of missing values on the activation of a neuron is neutralized . In this strategy , a missing value is not replaced by arbitrary values ( e.g. , through imputation ) ; it is naturally considered a missing piece of the puzzle ; we learn a problem-specific numerical representation for unknowns . The key feature of PROMISSING is its simplicity ; it is plug-and-play ; it deals with missing values in the first layer of the network without the need to change anything in the rest of the network architecture or optimization process . PROMISSING in its original form does not add extra parameters to the network , and its computational overhead remains negligible . Our experiments on simulated data and several classification/regression problems show that the proposed pruning method does not negatively affect the model accuracy and provides competitive results compared to several data imputation techniques . In a clinical application , making prognostic predictions for patients with a psychotic disorder , we present an application of PROMISSING on a multi-modal clinical dataset . We demonstrate how the NN model trained using PROMISSING becomes indecisive when facing many unknowns . This is a crucial feature for developing trustworthy prediction models in clinical applications . Furthermore , we show a side application of PROMISSING for counterfactual interpretation ( Mothilal et al. , 2020 ) of NNs decisions that can be valuable in clinics . 2 METHODS . Let x ∈ Rp represent a vector of an input sample with p features . We assume that the features in x are divided into two sets of q observed xo ∈ Rq and r missing features xm ( where p = q + r ) . In this study , we do not put any assumption on the pattern of missing values in x . Then , the activation of the kth ( k ∈ { 1 , 2 , . . . , s } ) neuron in the first hidden layer of an ordinary NN is : a ( k ) = ∑ xi∈xo xiw ( k ) i + ∑ xj∈xm xjw ( k ) j + b ( k ) . ( 1 ) This activation can not be computed unless the values in xm are imputed with real numbers . Here , in PROMISSING , we propose to alternatively replace the missing values with a neutralizer that 1 ) prunes the missing values from inputs of a neuron , 2 ) neutralizes the effect of missing values on the neuron ’ s activation by cancelling the second term in Eq . 1 and modifying the neuron ’ s bias . A missing value xj ∈ xm is replaced with its corresponding neutralizer u ( k ) j at the kth neuron , where : u ( k ) j = −b ( k ) pw ( k ) j . ( 2 ) The value of a neutralizer depends on its corresponding weight ( w ( k ) j ) , the bias of the corresponding neuron ( b ( k ) ) , and the number of features ( p ) ; thus , it can be computed on the fly during the training or inference procedures . A small value is added to weights before computing neutralizers to avoid division by zero . Inserting the neutralizer into Eq . 1 , the activation of the kth neuron is rewritten as : a ( k ) = ∑ xi∈xo xiw ( k ) i + qb ( k ) p , ( 3 ) in which the effect of weights of missing values on the activation of the neuron is eliminated , and the neuron ’ s bias is reduced by a factor of r/p . If all input values for a specific sample are missing then the neuron is completely neutralized . Proposition 1 If all input values are missing ( q = 0 and xo = ∅ ) then the activation of a PROMISSING neuron is zero ( see Appendix A.2.2 for the proof ) . Proposition 2 If there are no missing values in inputs ( q = p and xm = ∅ ) then the activation of a PROMISSING neuron is equal to a normal neuron ( see Appendix A.2.3 for the proof ) .. We should emphasize that , when using PROMISSING , the user does not need to apply any change to the input vectors , and the missing values ( generally represented as nans in the input matrix ) are fed directly to the network . After the training procedure , we eventually learn U ∈ Rs×p , a matrix representation for unknowns ( or metaphorically the dark matter ) : u ∗ ( k ) j = −b∗ ( k ) pw ∗ ( k ) j j ∈ { 1 , 2 , . . . , p } , k ∈ { 1 , 2 , . . . , s } , ( 4 ) where b∗ and w∗ are representing the final learned bias and weight . At the prediction stage , a missing value at jth input feature will be replaced with its corresponding neutralizer from U . It is worth emphasizing that the missing values are replaced with different neutralizers at different neurons ; therefore , it can not be considered a constant imputation . In fact , each neuron perceives differently a missing value in the input space . Metaphorically , the neurons can be seen as blind men in the parable of “ the blind men and an elephant ” 1 when facing unknowns . Furthermore , it is different from regression-based imputation and model-based approaches in the sense that a missing value in a specific feature is not inferred from other observed features , or the distribution of observed values ; i.e. , unknowns remain unknowns . In PROMISSING , we do not assume any certain missing value mechanism ( e.g. , MAR ) in advance . Instead , we try to learn the patterns of missing values from data that maybe advantageous in more difficult scenarios such as MNAR ( see results in Sec . 3.1 ) . One possible drawback of using PROMISSING is in high-dimensional input spaces and when the number of missing values is large , i.e. , when p → ∞ and r q . In this case , the neuron will undershoot ; hence the effect of few non-missing values are ignored . 2 To address this problem , we propose a modified version of PROMISSING ( mPROMISSING ) in which the effect of large r can be compensated with a compensatory weight , wc . The compensatory weight receives a fixed input of r/p for a specific sample ; thus , the activation of the neuron will change to : a ( k ) = ∑ xi∈xo xiw ( k ) i + qb ( k ) + rw ( k ) c p . ( 5 ) w ( k ) c is learned alongside the rest of the network parameters in the optimization process . On training data with few missing values , data augmentation ( e.g. , by simulating different patterns and size of missing values ) is advisable to ensure the sensibility of the learned compensatory weight . Since the input for this weight ( r/p ) is computed at the run time , no modification to input vectors is required . Proposition 3 If there are no missing values ( q = p , r = 0 , and xm = ∅ ) then the activation of an mPROMISSING neuron is equal to a normal neuron ( see Appendix A.2.4 for the proof ) . The proposed PROMISSING approach is straightforward to implement and use . It can be incorporated into the current implementations of different types of NN layers by adding/modifying few lines of code . We have implemented a nanDense layer , inheriting from Keras ( Chollet et al. , 2015 ) Dense layer , using PROMISSING and mPROMISSING neurons ( see appendix A.3 ) . The nanDense layer can be directly imported and used with any Keras model . In its general usage , the nanDense layer is only used for the first layer of an NN to handle missing values in inputs , unless we expect some missing values in the intermediate layers .

Summary: In this paper, the authors propose a method titled PROMISSING; this provides a new approach to handling missing data. Rather than imputation, a complete-case analysis, or inverse probability weighting, among other methods, the authors advocate for learning a problem-specific numerical representation for unknowns. The approach is interesting, and the experiments and data analyses are a good start at understanding the method.