A Modulation Layer to Increase Neural Network Robustness Against Data Quality Issues

1 INTRODUCTION . Despite the enormous academic and industrial interest in artificial intelligence , there is a large gap between model performance in laboratory settings and real-world deployments . Reports estimate that over 75 % of data science and artificial intelligence projects do not make it into production ( VentureBeat , 2019 ; Sagar , 2021 ; Chen and Asch , 2017 ) . One difficult transition from the laboratory is handling noisy and missing data . Errors in predictor data and labels ( Northcutt et al. , 2021 ) at the training stage are well understood to produce poor pattern recognition with any strategy ; garbage-in garbage-out . In the statistical learning literature , the effects of inaccurate and missing data on simple classifiers such as logistic regression is particularly well understood ( Ameisen , 2020 ) . As a result , datasets intended to train high-accuracy models are often carefully curated and reviewed for validity ( Ameisen , 2020 ; Xiao et al. , 2018 ) . However , when faced with noisy data from a new source , these models may fail ( L ’ Heureux et al. , 2017 ) . One special case is convolutional neural networks for machine vision ; augmenting the dataset with partially obscured inputs has been shown to increase the network ’ s ability to match low-level patterns and increases accuracy ( Zhong et al. , 2020 ) . No similar results with masking have been shown in tabular data , to our knowledge . These challenges are even more pronounced in applications that require high reliability and feature pervasive missing data at inference time , such as healthcare ( Chen and Asch , 2017 ; Xiao et al. , 2018 ) . Electronic health records ( EHR ) can contain a high percentage of missing data both at random ( keyboard entry errors , temporarily absent data due to incomplete charting ) and informative or missing-not-at-random ( MNAR ) data ( selective use of lab tests or invasive monitors based on observed or unobserved patient characteristics ) . Medical measurements also have non-uniform noise ; for instance , invasive blood pressure measurement is more accurate than non-invasive blood pressure ( Kallioinen et al. , 2017 ) . Another example is the medical equipment by different manufacturers that have various margins of error which affects the accuracy and hence the reliability of the measurement ( Patel et al. , 2007 ) . Mammalian brains have a distinct strategy to integrate multi-modal data to generate a model of the surrounding environment . They modify the impact of each input based on the presence and reliability of other signals . This effect can be observed dynamically in response to temporary changes in available inputs ( Shine et al. , 2019 ) , as well as long-term as a compensation mechanism for permanent changes such as neural injuries ( Hylin et al. , 2017 ) . For example , a human brain gives less weight to visual input in a dark environment and relies on prior knowledge and other sensory cues more . Unlike simply down-weighting low-accuracy data , replacement data with related information is up-weighted . This is usually modelled as a Bayesian inference process ( Cao et al. , 2019 ; Ernst and Bülthoff , 2004 ; Alais and Burr , 2004 ; Heeger , 2017 ) . This modulation of different input is also observed in other organisms where the neural behavior of a neuron or a group of neurons can be altered using neuromodulators ( Harris-Warrick and Marder , 1991 ) . We used the inspiration from this process to design a fully-connected neural network layer with variable weights . Those weights could be modulated based on a variety of inputs , but we focus on input reliability as a modulating signal . This allowed us to train the neural network using datasets that are loosely preprocessed with a high incidence of missing data while achieving high performance . At inference time , the network is capable of producing accurate outputs despite signal degradation . A restricted structure of modulating inputs and effects on the modulated layer reduces the likelihood of severe over-fitting and complexity of the estimation problem . 2 RELATED WORK . The most obvious use case we propose for this structure is handling missing data . There is a vast literature on imputation , which also attempts to use alternative inputs to replace missing data . Classical simple methods of imputation include constant values ( e.g . mean imputation ) , hot deck , k-nearest neighbor , and others ( Buck , 1960 ) . Single or multiple imputation using chained equations ( Gibbs sampling of missing data ) is popular due to its relative accuracy and ability to account for imputation uncertainty ( Azur et al. , 2011 ) . More advanced yet classic methods have seen relative success such as Bayesian ridge regression ( MacKay , 1992 ) and random forest imputation ( Stekhoven and Bühlmann , 2012 ) . Deep learning-based imputation has been used recently using generative networks ( Beaulieu-Jones and Moore , 2017 ; McCoy et al. , 2018 ; Lu et al. , 2020 ; Lall and Robinson , 2021 ; McCoy et al. , 2018 ; Mattei and Frellsen , 2019 ; Yoon et al. , 2018 ; Ivanov et al. , 2019 ) and graph networks ( You et al. , 2020 ) . Our modulation approach can be incorporated into autoencoder architectures to improve their performance and stability in data imputation , but it also provides the flexibility of skipping the imputation step altogether when the task performed does not require imputation ( i.e . classification ) thus skipping a preprocessing step and saving processing time . Incorporating uncertainty measurements into deep neural networks has also been approached with Bayesian deep learning methods , ( Wang and Yeung , 2016 ; Wilson , 2020 ) which has a complex , assumption laden structure using probabilistic graphical models . One simpler variation of Bayesian deep learning is the Gaussian process deep neural network which assigns an uncertainty level at the output based on the missing data so that inputs with greater missingness lead to higher uncertainty ( Bradshaw et al. , 2017 ) . Our method makes use of meaningful missingness patterns as opposed to treating it as a problem that leads to lower confidence in outputs . Our approach is superficially similar to attention mechanisms : the lower quality inputs receive less importance , but attention networks employ a complex feedback mechanism to assign the attention distribution using the input sequence and the query and is thus difficult to scale for long time-varying inputs ( Kim et al. , 2017 ) . 3 METHODS . 3.1 ARCHITECTURE . A fully connected layer has a transfer function of hout = f ( W · hin + b ) , ( 1 ) where hin is the input to the layer , W is the weight matrix , b the bias and f the non-linearity function . W is optimized during training and fixed at inference . We propose a modulated fully connected layer ( MFCL ) where weights are made variable by replacing W by Wmod ( Figure 1 ) where Wmod = g ( m ) , ( 2 ) where m is the modulating signal input and g is the function that is defined by a multilayer perceptron . Another variant of the MFCL ( MFCL+ ) adds a skip connection to the transfer step and modifies the weights ( W0 ) of a starting network , Wmod+ = g ( m ) +W0 . ( 3 ) The latter architecture aims at being more adaptive to datasets with less or no missing values and could avoid potential instabilities by having the entire layer weights variable . 3.2 EXPERIMENTS We assessed the performance of the MFCL and MFCL+ layers in classification , regression , and imputation tasks . These experiments used modulating signals of missing value flags and input reliability values of noisy data . We can think of missing values as a special case included in reliability where missing implies completely unreliable measurement . For the sake of clarity , we test the cases of missing values and noisy values separately rather than combining them . For baseline comparison , we employed models with matching architectures while swapping the first fullyconnected layer with a MFCL/MFCL+ . Base architectures were guided by previous best performing models in the literature . Modulation network architectures were optimized using a grid search . A complete description of the architectures is elaborated in the appendix . 3.3 DATASETS . The motivating dataset for our experiments derives from operating room data from Barnes Jewish Hospital ’ s Anesthesiology Control Tower project ( ACTFAST ) spanning 2012–2018 . The [ Name redacted for anonymity ] IRB approved this study and granted a waiver of informed consent . The dataset contains preoperative measurements of medical conditions , demographics , vital signs , and lab values of patients as well as postoperative outcomes that were used as labels for supervised learning including 30-day mortality , acute kidney injury , and heart attack . The ACTFAST dataset was used in previous studies for prediction of 30- day mortality ( Fritz et al. , 2019 ) , acute kidney injury , and other complications ( Cui et al. , 2019 ; Abraham et al. , 2021 ; Fritz et al. , 2018 ) . For predictors , we utilized a subset of the input features of preoperative vital signs and lab values ( 15 variables ) . Table 1 ( appendix ) shows a list of variables used and the missing percentages . Table 2 ( appendix ) shows the distribution of outcome values , which have a large imbalance between positive and negative samples . We also used the Wisconsin Breast Cancer dataset for classification of tumors from features extracted from a fine needle aspirate of breast mass image ( Mangasarian et al. , 1995 ) . We used the Boston housing prices dataset ( Harrison Jr and Rubinfeld , 1978 ) as a regression example . Each of the above datasets was also used for imputation tasks . 3.4 CLASSIFICATION TASK . We ran five experiments for classification using the ACTFAST and Breast Cancer datasets . Four experiments utilized the missing flags as a modulating signal ; in the last experiment , we utilized input reliability as a modulating signal . Reliability was quantified by the standard deviation of the noise that was artificially added to the signal . We tested the MFCL and MFCL+ in the place of fully-connected ( FC ) layers at the input level . 3.4.1 BASELINES . The baseline classifiers for the ACTFAST and Breast Cancer datasets were four MLPs with matching hidden layer structures that were fed imputed values using different algorithms , namely : chained regression with Bayesian ridge regularization ( Scikit Learn Iterative Imputer ) , missForest algorithm ( Stekhoven and Bühlmann , 2012 ) , GAIN ( Yoon et al. , 2018 ) , and VAEAC ( Ivanov et al. , 2019 ) . We also tested one graphical network model GRAPE ( You et al. , 2020 ) . The last model applied constant value imputation ( mean value ) in addition to concatenating an indicator variable for missing values at the input layer ( FC+Mod ) . ACTFAST We built three classifiers to predict 30-day Mortality , Acute Kidney Injury ( AKI ) , and Heart Attack from the preoperative input features ( Table 1 ) . We used the datasets with the inherent missing data for training and then tested the trained models with additional missingness artificially introduced in both random and non-random fashions . Non-random missingness was introduced by removing the largest values and by all datapoints of features ( discussed below ) . Breast Cancer For the classifier with missing flags as modulating signal , we introduced nonrandom missingness into the training dataset by removing the highest quartile of each variable . At the testing phase we evaluated each model with additional missingness similar to the ACTFAST classifiers . For the classifier with reliability signal , we utilized the complete dataset but added Gaussian noise with zero mean and variable standard deviation ( SD ) where the SD values were sampled from a uniform distribution between 1 and 10 standard deviations of each variable . The higher end of SD values is very large , simulating a spectrum of noisy to essentiall missing data . Breast cancer dataset includes values that were measured from images of a fine needle aspitate of a breast mass to describe cell nuclei characteristics . Extracted data includes mean values as well as standard errors and worst value measurements . Due to this nature of variables and them including error rate values , we selected 10 variables representing the mean value measurements only for this experiment . We tested using a 20 % test split on the same noisy data .

Summary: The paper is proposing a neural net architecture which makes Weights (in a layer) a function of an external signal trying to quantify the quality of input. The presented architecture is motivated by, and applied to the problem of missing/noisy data. Empirical evaluation performed on different types of tasks, over different datasets and in different missing-ness scenarios showed improved robustness in terms of predictive performance.