PROMISSING: Pruning Missing Values in Neural Networks
Authors: Seyed Mostafa Kia
Nastaran Mohammadian Rad
Daniel van Opstal
Bart van Schie
Wiepke Cahn
Andre Marquand
Josien P.W. Pluim
Hugo G. Schnack
Venue: ICLR 2022 Submitted (2022)
Keywords: Neural Networks
Missing Values
Data Imputation

Abstract: While data are the primary fuel for machine learning models, they often suffer from missing values, especially when collected in real-world scenarios. However, many off-the-shelf machine learning models, including artificial neural network models, are unable to handle these missing values directly. Therefore, extra data preprocessing and curation steps, such as data imputation, are inevitable before learning and prediction processes. In this study, we propose a simple and intuitive yet effective method for pruning missing values (PROMISSING) during learning and inference steps in neural networks. In this method, there is no need to remove or impute the missing values; instead, the missing values are treated as a new source of information (representing what we do not know). Our experiments on simulated data, several classification and regression benchmarks, and a multi-modal clinical dataset show that PROMISSING results in similar classification performance compared to various imputation techniques. In addition, our experiments show models trained using PROMISSING techniques are becoming less decisive in their predictions when facing incomplete samples with many unknowns. This finding hopefully advances machine learning models from being pure predicting machines to more realistic thinkers that can also say "I do not know" when facing incomplete sources of information.

Under review as a conference paper at ICLR 2022
PROMISSING: PRUNING MISSING VALUES IN NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
While data the primary fuel for machine learning models, they often suffer from missing values, especially when collected in real-world scenarios. However, many off-the-shelf machine learning models, including artificial neural network models, are unable to handle these missing values directly. Therefore, extra data preprocessing and curation steps, such as data imputation, are inevitable before learning and prediction processes. In this study, we propose a simple and intuitive yet effective method for pruning missing values (PROMISSING) during learn- ing and inference steps in neural networks. In this method, there is need to remove or impute the missing values; instead, the missing values are treated as source of information (representing what do not know). Our experi- ments on simulated data, several classification and regression benchmarks, and multi-modal clinical dataset show that PROMISSING results in similar predic- tion performance compared to various imputation techniques. In addition, our experiments show models trained using PROMISSING techniques are becoming less decisive in their predictions when facing incomplete samples with many un- knowns. This finding hopefully advances machine learning models from being pure predicting machines to more realistic thinkers that can also say "I do not know" when facing incomplete sources of information.
INTRODUCTION
Missing and incomplete data are abundant in real-world problems; however, the learning and infer- ence procedures in machine learning (ML) models highly rely on high-quality and complete data. Therefore, it is necessary to develop new methods vironments. Currently, the most popular way to deal with imperfect data is to impute the missing values. However, if consider the learning and inference procedures in our brain as a role model for ML algorithms, data imputation barely follows the natural principles of incomplete data process- ing our brain. This is because the imputation is generally based on using a heuristic for replacing missing values. Our brain does not impute incomplete sensory information but instead uses its in- completeness as separate source of information for decision making. For example, by only hearing the rain we can estimate how hard it is raining, and we do not necessarily need to receive visual in- formation. Instead, we direct our attention more toward our auditory inputs to decide whether to go with umbrella. addition, the more we miss sensory information, the more cautious we get in decision-making. That is why we are more careful in darker environments.
deal with data imperfections in rugged en-
Neural networks (NNs) are brain-inspired algorithms that very popular these days (under the name of deep learning) for learning complex relationships between inputs and target variables. How- ever, they are in principle unable handle incomplete data with missing values. They mainly rely on matrix operations which cannot operate on not-a-number (NaN) values. Only one NaN in a dataset impairs the forward propagation in a network. There are three solutions to this problem (Gareía- Laencina et al.] 2010): i) removing samples or features with missing values, ii) imputing the miss- ing values, and iii modeling the incomplete data. Removing the samples with missing values can be very costly, especially in small-sample size and high-dimensional datasets. For example, data collection in clinical applications is an expensive procedure in terms of time, finance, and patient burden. Moreover, removing even few samples from small datasets can affect negatively the gener- alization performance of the final model. Removing informative features with missing values is also compensated with lower model performance. Therefore, filling the information gaps is inevitable.

---

Under review as a conference paper at ICLR 2022
There are various techniques for data imputation, ranging from simply imputing missing values with a constant to more sophisticated ML-based imputation approaches (Little & Rubin 2019: Gareía- Laencina et al., 2010). One can categorize the most common techniques into three main categories: 1) constant imputation, ii) regression- imputation, and iii) MI--based imputation. In constant imputation, the missing values are replaced with constant, e.g., zeros mean/median of features. It has been shown that constant imputation is Bayes consistent when the missing features are not informative (Josse et al. 2019). In the regression-based imputation, linear or non-linear regression model is derived to predict the missing values. This method can be used to impute a single or multiple features. The most popular Equations (MICE) (Van Buuren & regression- based imputation is Multiple Imputation by Chained Outdhooom 2011] Azur et al.] 2011). In MICE, an iterative procedure of predicting missing values and re-training regressors with updated predictions is performed for a limited number of cycles. The central assumption behind the MICE approach is that the missing values are missed at random (see Rubin (176)) and Appendix [. for definitions of missing value mechanisms including missing completely at random (MCAR), missing random (MAR), and missing not at random (MNAR)). Applying MICE can result in biased estimations if this assumption is not satisfied (Azur et al. 2011). Another critical limitation of regression-basee imputation is its high computational complexity (Caiafa et al. 2021). If we do not know which feature will be missed the test time, for d features, we need train d different regression models.
In the ML-based approach ML algorithms, such as a K-nearest neighbor (KNN), regularized linear
model (Jaag et al. (021)) decision trees (Twala et al. 2008). random random forest (Xia et al.
2017). neural 2020 Collier
network (Bengio & Gingras 1996), or generative model (Yoon et al. 2018 Ipsen Ipsen et al.
et al. 2020 Nazabal et al. 2020), are used for handling missing data.
As an alternative solution to data imputation, one can use the elegance of probabilistic modeling to model the incomplete data under certain assumptions. One seminal work in this direction is presented by GGhahramani & Jordan (1994). where a Gaussian Mixture Model (GMM) is used to estimate the joint density function on incomplete data using an Dnnectattoo-Maxiiizzan
(EM) al- gorithm. This approach is later adopted and extended to logistic regression (Williams et al. 2005), Gaussian processes, support vector machines (Smola et al. (005). and multi-class non- non-linear clas- sification (Liao et al. 2007). However, despite their good performance
small-size datasets, their application remained limited on big and high-dimensional data due to the high computational com- plexity (Caiafa et al., 2021). To overcome this issue, [Caiafa al. (202T) proposed sparse dic- tionary learning algorithm that is trained end-to-end, and simultaneously learns the parameters of the classifier and sparse dictionary representation. [e Morvan al.] (020) proposed NeuMiss, network architecture that uses differentiable imputation procedure in a theputeethe--rgress scheme (Morvan et al.) (202T). A notable feature of NeuMiss is its robustness to MÑAR data. Inverse probability weighted estimation (Wooldridge) 2007 /Seaman & White 2013) is another probabilis- tic approach for handling missing values without imputation in which the weights for samples with many missing values are inflated based an estimation of the sampling probability. Recently, Smieja et al. (2018) proposed a modified neuron structure that uses GMM with a diagonal covari- ance matrix (assuming MAR) to estimate the density of missing data. GMM parameters are learned with other network parameters. Conveniently, it handles missing values in the first layer of the network, and the rest of the architecture remains unchanged. Elsewhere, Nowicki et al. (2016) pro- posed a new neural network architecture based on rough set theory (Pawlak) 1998) for learning from imperfect data. It is fascinating that this method can say "I do not know" when large portion of input values are missing, unlike traditional models trained on imputed data that may predict definite outcomes even on completely unmeasured samples, i.e., they run in the absolute darkness. These predictions can be dangerous with catastrophic consequences in more delicate applications of ML for example in autonomous driving, robotic surgery, or clinical decision- decision-making.
In this work, we attack the problem of modeling incomplete data using artificial neural networks without data imputation. We propose simple technique for pruning missing values (PROMISS- ING) in which the effect of missing values on the activation of a neuron is neutralized. In this strategy, a missing value is not replaced by arbitrary values (e.g., through imputation); is naturally considered a missing piece of the puzzle; we learn problem-specific numerical representation for unknowns. The key feature of PROMISSING is its simplicity; it is plug-and-play; it deals with missing values in the first layer of the network without the need to change anything in the rest of the network architecture or optimization process. PROMISSING in its original form does not add extra parameters to the network, and its computational overhead remains negligible. Our experiments on simulated data and several classfiication/eegesssoon problems show that the proposed pruning
2

---

Under review as a conference paper at ICLR 2022
method does not negatively affect the model aceuracy and provides competitive results compared to several data imputation techniques. In a clinical application. making progmostic predictions for patients with psychotic disorder, present application of PROMISSING on multi- clinical dataset. We demonstrate how the NN model trained using PROMISSING becomes indeci- sive when facing many unknowns. This is crucial feature for developing trustworthy prediction models in clinical applications. Furthermore, we show a side application of PROMISSING for coun- terfactual interpretation (Mothilal al. 2020) of NNs decisions that can be valuable in clinics.
METHODS
Let E IR" represent a vector of an input sample with p features. We assume that the features in æ are divided into two sets of observed x" E IR and missing features x"' (where p + r). In this study, we do not put any assumption on the pattern of missing values in æ. Then, the activation of the kth (k E {1, , x)) neuron in the first hidden layer of an ordinary NN is:
(R))
Σ SE
Σ ZyE
(K) kyW + 1((0
TAW
This activation cannot be computed unless the values in xx" imputed with real numbers. Here, in PROMISSING, we propose to alternatively replace the missing values with neutralizer that 1) prunes the missing values from imputs of neuron. 2) neutralizes the effect of missing values on the neuron's 's activation by cancelling the second term in Eq. and modifying the neuron's 's bias. A missing value æz E x replaced with its corresponding neutralizer uAA the kth neuron, where:
(())
(k)
pro (R)
The value of a neutralizer depends on its corresponding weight (wo))), the bias of the corresponding neuron (b), and the number of features (p): thus, it can be computed the fly during the training or inference procedures. A small value is added to weights before computing neutralizers to avoid division by zero. Inserting the neutralizer into Eq. [ the activation of the kth neuron rewritten as:
gb(k) P
(k)
Σ 27's Exr
in which the effect of weights of missing values on the activation of the neuron is eliminated, and the neuron's bias is reduced by a factor r/p. all input values for specific sample are missing then the neuron is completely neutralized.
Proposition 1 If all input values are missing (g = 0 and x = Ø) then the activation of a PROMISSING neuron is zero (see Appendix A.2.2 for the proof).
Proposition 2 there are no missing values in inputs (q and x 2 then the activation of PROMISSING neuron is equal to a normal neuron (see Appendix A.2.3 for the proof)..
We should emphasize that, when using PROMISSING, the user does not need to apply any change to the input vectors, and the missing values (generally represented as nans the input matrix) are fed directly to the network. After the training procedure, we eventually learn U E Rxp matrix representation for unknowns (or metaphorically the dark matter):
(A)
(N)
j (L. ph. k (1.
(4)
pwy (K)
where 6" and w" are representing the final learned bias and weight. At the prediction stage, a missing value at jth input feature will be replaced with its corresponding neutralizer from U. is worth emphasizing that the missing values replaced with different neutralizers at different neurons; therefore, it cannot be considered a constant imputation. In fact, each neuron perceives differently missing value in the input space. Metaphorically, the neurons be as blind men in the
3

---

Under review as a conference paper at ICLR 2022
parable of "the blind men and elephant" when facing unknowns. Furthermore, it is different from regression- based imputation and model-based approaches in the sense that a missing value in specific feature is not inferred from other observed features, the distribution of observed values; Le. unknowns remain unknowns. In PROMISSING, we do not assume any certain missing value mechanism (e.g., MAR) in advance. Instead, try learn the patterns of missing values from data that maybe advantageous in more difficult scenarios such as MNAR (see results in Sec. 3.1
One possible drawback of using PROMISSING is in high-dimensional input spaces and when the number of missing values is large, i.e., when p + oo and r > q. In this case, the neuron will undershoot; hence the effect of few non-missing values are ignored. [ To address this problem, we propose a modified version of PROMISSING (MPROMISSING) in which the effect of large r can be compensated with compensatory weight, We- The compensatory weight receives a fixed input of r/p for a specific sample; thus, the activation of the neuron will change to:
2, gb(*) rwe (R)
a
(5)
P
E
we is learned alongside the rest of the network parameters in the optimization process. On training data with few missing values, data augmentation (e.g., by simulating different patterns and size of missing values) is advisable to ensure the sensibility of the learned compensatory weight. Since the input for this weight (r/p) is computed at the run time, no modification to input vectors is required.
Proposition 3 If there are no missing values (q = p. = mPROMISSING neuron is equal to a normal neuron (see Appendix Appendix(A..A4 A.2.4
= 0, and xr" = (3) then the activation of an the proof).
The proposed PROMISSING approach is straightforward to implement and use. It can be incor- porated into the current implementations of different types of NN layers by adding/modifying few lines of code. We have implemented a nanDense layer, inheriting from Keras (Chollet et al.) 2015) Dense layer, using PROMISSING and mPROMISSING neurons (see appendix [A.3). The nanDense layer can be directly imported and used with any Keras model. In its general usage, the nanDense layer is only used for the first layer of an NN to handle missing values in inputs, unless we expect some missing values in the intermediate layers.
3 EXPERIMENTS AND RESULTS
In a set of three experiments, 1) in a simulation study, we investigate the convergence behaviors of (m)PROMISSING on MCAR, MAR, and MNAR data; 2) in large experiment on real data, benchmark the reliability of (m)PROMISSING in several classification and regression tasks, and 3) on a multi-moda, clinical dataset, we demonstrate an application of PROMISSING on an open problem of psychosis prognosis prediction.
3.1 SIMULATED DATA
We conducted a simple analysis of synthesized data to better understand the behaviors of PROMISS- ING. For data simulation, we simulated the XOR problem with 1000 samples. Gaussian noise with a variance of 0.25 is added to data. We simulated the missing data for MCAR, MAR, and MNAR settings. The same procedures
in Schelter et al. (2021) are used to simulate the MCAR, MAR, and MNAR missing values (see Appendix AA.4 for details). The experiments are repeated for 30% and 50% missing samples. We used scikit-learr (Pedregosa et al. (2011) for implementing different data imputation schemes We also compared our method with a GAN- imputation proposed in Yoon al. (0118). Default configurations used for all imputers.
To evaluate PROMISIIIG we used a simple fully connected NN architecture with four tangent- hyperbolic neurons in the hidden layer and a single sigmoid neuron in the output layer. This simple architecture can reach the performance of the Bayes optimal classifier (AUC=0.99 99 + 0.01) on simu- lated data without missing values (see the black lines in Fig. D.. We randomly divided the simulated
'ee lattps: //en. wik wikipedia..coo www.diia.rrg/wwkieee http//www////iireeee men and an elephant for the parable. 2 Nooee that this again might he considered as an advantage in some applications.
"Scikit-lcam 0.24.2 with Numpy 1.19.5.
4

---

Under review as a conference paper at ICLR 2022
MCAR
MAR
MNAR
1.0
0.8
0.6
PECE
GANN
FROMESSIN
0.4
IIPRORISSENG
1.0
0.6
0.4
200
400
600
800
1000
200
400
600
800
1000
200
400
600
800
1000
Epochs
Epochs
Epochs
Figure Comparison between learning curves of PROMISSING and mPROMISSING with data imputation approaches on simulated MCAR, MAR, and MNAR data, when tested a test set without missing values (first row) and with 50% missing values (second row).
data into training and test sets with 500 samples in each set. The pipeline of 1) data simulation, 2) random data corruption with missing values, 3) randomly splitting data into the training and test set, and 4) training the model is repeated 100 times, and the standard deviation of performances across repetitions is reported (shadowed areas in Fig. I We have fixed the random number generator seed in each repetition (randomly selected from [0, I00000]) to ensure a fair comparison between mod- els. A stochastic gradient descent (SGD) optimizer is used minimize binary cross-entropy loss function. We evaluated the network performance as the training procedure proceeds in two different settings i) on the complete
without missing data, and ii) on the
with missing values.
The first row of plots in Fig. D shows the area under the ROC curve (AUC) for models trained on data with 50% missing values (see Fig. 7 for 30%) and tested on sets without missing values as the number of epochs progresses. Here, by comparing the performance gap between the different missing data curation strategies and the optimal Bayes classifier, we can evaluate the training bias imposed by each strategy during the training. A lower training bias indicates lower effect of data curation strategy on the quality of the trained model. Our empirical results show that the training bias is modulated by both the strategy and missing data mechanism. In the MCAR setting, all strategies show less training bias. However, in the MAR and MNAR conditions, the model biases more pronounced, especially with higher percentages of missing values. The (m)PROMISSING techniques show a competitive performance, especially in the MNAR setting where they perform equivalently with mean, KNN, and MICE imputers (with Wilcoxon rank-sum test p-values of 0.26, 0.02, and 0.15) and better performance than zero and GAIN imputations (p-valuess x 10-12). These results confirm that the (m) PROMISSING models provide the possibility learn meaningful representations for the unknowns that are at least as informative as imputed data.
The second row of plots in Fig. [ compares the learning curves of models when tested on the test set with missing values. By comparing the final performance of models in the first and second row, we immediately notice the performance drops across almost all strategies. These performance drops show the negative effect of data degradation imposed by each imputation method on the generalization of models when applied data with missing values. Thus, we refer this dif- ference as test bias. While all methods show similar test bias in the MCAR and MAR scenarios, the (M)PROMISSING models show a negligible test bias in the MNAR scenario, i.e., the performance of (m)PROMISSING models is not affected when they applied to test data with MNAR missing values. Furthermore, m(PROMISSING) methods show significantly better performance on MNAR test data (p-values> 6 10-6). The small training and test biases of the (m)PROMISIING models in the MNAR setting confirm their effectiveness in jointly modeling the data and missing patterns.
We have further analyzed the patterns of neutralizers in U to interpret the learned representations of unknowns in PROMISSING (see Fig. X in appendix A.5). Since U derived from the network parameters, due to the stochastic nature of parameter initialization and optimization, it may end
5

---

Under review as a conference paper at ICLR 2022
MCAR
MAR
MNAR
0.08
0.06
0.06
AUC
0.04
0.04
0.02
0.02
0.02
0.10
0.25
0.50
0.90
0.10
0.25
0.50
0.90
0.10
0.25
0.50
0.90
Sample Nissing Rate
Sample Missing Rate
Sample Missing Rate
Figure 2: Comparing AUC drops in classification tasks with respect to the full model across five different imputation methods, HGB, and (m)PROMISSING in MCAR, MAR, and MNAR settings.
up with different values in each repetition. Our visual inspection shows that each neuron learns a different solution for unknowns, ranging from a simple zero or mean imputer to more complex patterns. This is unique feature for PROMISSING providing the possibility to learn neuron- specific imputation strategy from data.
3.2 OPENML DATA
In the second set of experiments, we benchmarked (m)PROMISSING alongside several impu- tation techniques on a range of publicly available classification and regression datasets from OpenML (Vanschoren et al., 2014). We used the same classification and regression datasets as in Jäger et al. (2021) (see Table and Table 3 in appendix (A.6). Here, we use simple fully- connected architecture with two hidden layers, with p/2 ReLU neurons in the first and two neurons in the second hidden layer, so the architecture changes slightly from one dataset to another depend- ing on the number of features. A single output neuron with a sigmoid or linear transfer function is used for output layers in classifiers or regressors, respectively. Our quick experiments on bench- mark datasets show that this simple architecture provides competitive results with random forest classifiers and regressors (see Fig T0 in appendix A.6). We excluded three classification and three regression datasets from our analyses due to the poor performance of baseline models on the com- plete data (see appendix A.6). We used MCAR, MAR, and MNAR schemes to simulate different ratios (0.1, 0.25, 0.5, 0.9) of samples with missing values across datasets. The same procedures as in Schelter al. (2021) are used simulate the MAR and MNAR missing values. In the MNAR setting, random percentile of the most informative feature (based on the mutual information with the target variable) is removed. In the MAR case, values in the most informative feature are removed based on random percentile of the second most informative feature (see Appendix A.4 for more details). An SGD optimizer for 100 epochs with mini-batch size of 10 samples used in the training process to minimize a binary cross-entropy or mean squared error (MSE) loss in the clas- sification or regression scenario. The performances of different methods are evaluated in a 2-fold cross- volsdavali scheme computing the AUC and standardized MSE (SMSE) metrics. The whole experiment pipeline, including 1) simulating missing values, 2) data imputation (this step does not apply to (m)RROMISI)G)) 3) data standardization, twenty times with a fixed random number generator seed in each repetition. When imputing data, the default imputer settings are applied. We have also included the histogram-based gradient boosting (HGB) (e et al.) 2017) -that similar to (M)PRIIISSING provides a mechanism to directly deal with missing values without imputation- in our experiments.
model training and evaluation, repeated
Fig. compares the performance of different data imputation methods with (M))RMSISIING in classification tasks for various missing value mechanisms and missing sample ratios (x (x-axes). The AUC drops with respect to the full model (trained and tested on complete data) are plotted. To sum- marize the results across datasets and replication runs, we first compute the median performance across datasets and then average across twenty replications. While in all settings the HGB is the best performer, the (M)SSSSISI shows competitive performance with the better performing imputers. A similar overall pattern can be observed when comparing the increase in the SMSE in re- gression tasks (see Fig. [ in appendix A.6). In summary, these results show that PROMISSING can compete with data imputation approaches without the need for filling the unknowns, thus it is more flexible with underlying missing data mechanisms and all imput data types including continu- ous, binary, and categorical.

---

Under review as a conference paper at ICLR 2022
3.3 CLINICAL APPLICATION: PSYCHOSIS PROGNOSIS PREDICTION
Missing data are seemingly ubiquitous in the healthcare domain. In general, clinical datasets contain multi-modal data that are collected through different measurement tools, including patient question- naires, medical reports and instruments, biological tests, and imaging data. Each modality may represent a different aspect of a disease or treatment outcome. Therefore, employing smart modality fusion techniques is inevitable to best benefit from these rich data. However, this goal is occluded be- cause these data are collected through different sources and often include missing variables or even missing modalities. In this section, we demonstrate application of PROMISSING in multi-modal data fusion and counterfactual model interpretation for psychosis prognosis prediction (PPP).
3.3.1 OPTIMISE DATASET
We used the OPTiMiSE dataset (Kahn et al.) (2018), an antipsychotic three- phase switching study: 495 patients with schizophrenia, schizophreniform, or schizoaffective disorder according to DSM-IV diagnosis standards. The pa- tients received different medications in three-phase study design. We used the data from the first phase of the study in which the patients were treated with amisulpride for four weeks. Only 376 patients who finished the first phase are used in our experiments. a PPP framework, we aimed to predict the probability of patients' symptomatic remission based on the Positive and Negative Syndrome Scale (PANSS) as defined by consensus criteria (Andreasen al.) 2005). We used 12 different data modalities in our experiments (see Table [ In total, 13% values in the dataset are missing. These missing values include complete or partial missed modalities. The missing values scattered across patients and measures; somehow, if we remove all subjects with missing values only 17 patients are left. This observation substantiates, even more, the importance of missing data management in clinical datasets. We have applied minimal preprocessing pipeline the data before the modeling stage. The input data contains binary, categorical, and continuous variables. We used 0/1 and one-hot encoding for binary and categorical variables, respectively. All continuous variables are standardized before feeding them to the model. Considering the unbalanced class dis- tributions (67% of patients get remitted), in each fold of K-fold -ols-valii,,,on, resampled the minority class samples in the training data to match its size to the majority class.
Table I: Data modalities in the OPTiMiSE dataset.
Medbaity
Meneraa
Somptks NuNo
Percentags
Desrtpptin
Denegraphic
Seca- ccssgradii Ecearc
LiGstek
421
2110
Liee of schiawwo like drap and alookel
67
14%
do audicttion
MINI
Tho chaili comsehiiit
99/5
PANSS
Pedliic and Napate Syradiene Seale
Clisicll Cikeaal laspraskes
CESS
1115
Mosare siale whee deposssion
Summnory
1025
3.3.2 A MULTI-MODAL MODEL FOR PPP
Inspired by Huang al. 20200, we used multi-modal NN architecture with middle and
Probuisity of Ramasdon
late information fusion. As depicted in Fig. 3 model has five layers: input, representa- tion learning, modality-specific classification, fusion, and classification. The network re-
Clansication Layer
Dhniherr
Fusiun Layez
Derse
ceives the data from M different modalities in the input layer. Since input samples may con- tain missing values, we use nanDense lay- ers right after the input layer in the repre- sentation learning phase. These layers trans- fer the raw inputs to an intermediate repre- sentations. We roughly (without any opti- mization and solely based on input feature size) fixed the number of neurons in this layer for each modality (Demograpphcs==10, Diagno- sis=10, Lifestyle=5, Somatic=5, Treatment=2, MINI=10, Cytokines=10, PANSS=10, PSP=5, CGI=2, CDSS=5, SWN=5). In the modality-spccific classification layer, use 2 Softmax neurons
Casoffiction Layers
Choufee
Cusifir
Representation Learning
nenDerse 2
zanDlene2 2
ranDenne
Imper Layees
Modility
Modility
Modally M
Figure 3: The NN architecture with middle and late information fusion for psychosis prognosis prediction on multi-modal data.
7

---

Under review as a conference paper at ICLR 2022
(a) IMPUTATION
(b) PROMISSING
(c) MPROMISSING
Probability Remission
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
2
10
0
10
12
12
Number of Missing Modality
Number of Missing Modelity
Number of Missing Modality
Figure 4: The trajectories of model predictions for 17 patients with information loss when using a) data imputation, b) PROMISSING, and c) MRPOMISSING methods to handle missing data.
to classify the -opelity-speciffi representations into target classes, i.e., whether patient remits given a certain amount of medication after four weeks. These decisions are merged with middle representation and then fed the fusion layer. A trivial ReLU Dense layer with five neurons is used for the fusion layer. Finally, two Softmax neurons are used to classify the fused data into target classes. Dropout layers with probability of 0.1 are applied before any weighted layer. An Adam (Kingma & Bal 2014) optimizer (with learning rate of 0.0003) is used to minimize the total categorical cross-entropy loss across classification layers.
3.3.3 EVALUATION AND COMPARISON
We first compared (m) PROMISSING with two alternative strategies: i) removing features with miss- ing values and ii) data imputation. In the first case, we first removed subjects that completely missed one data modality; then, we removed features containing missing values. This procedure ended up losing 67 subjects and 78% of features including two modalities. In the second case, we used a KNN to impute the missing data. Note that, due to binary and categorical features, many imputa- tion techniques (such as mean and iterative imputer) cannot be directly applied to our data. Even in the KNN case, we needed to set the number of neighbors equal to one to avoid float numbers for the categorical features. A 10-fold cross- validaaaii scheme is employed for model evaluations, and all the experiments are repeated ten times to evaluate the variability in performances. The re- sults support our conclusion in Sec. 3.2 that (MIRRISISINGG performs well data imputation. PROMISSING and mPROMISSING result in AUCs of 0.67 + 0.02 and 0.66 + 0.02 that are on a par with the KNN imputer 0.66 + 0.02. However, is expected, removing missing features resulted in an impaired model performance of 0.61 + 0.02. The similar performance of PROMISSISN and data imputation despite their completely different strategies to deal with missing values raises two questions: "Do the resulting models behave similarly too?", "If not, how are they different?" Craving to answer these questions, we set up the second set of experiments on the OPTiMiSe data.
In the next experiment, we left all the 17 patients with complete data in the test set and used the other 359 patients (with missing data) for training. Here, to fully use the capabilities of PROMISSINGG models, we have augmented the training data by removing different subsets of modalities from data. Having M 12 modalities and n = 470 samples (after balancing the training data) in the training set, this procedure resulted in 1, 919, 010 samples (EM-1 (M) x n.) We trained three models, one using KNN imputation and two using (m) PROMIISSING Then, have used a particular procedure for testing these models on the test set. In 100 repetitions, we have removed the modalities one by one and in random order. Fig. shows the trajectories of predictions for 17 test subjects across three models. The trajectories show the predicted probability of remission when the model has access to less information about the patients (from complete information on the left to no information on the right). Note that it is impossible to remove all modalities in the KNN imputation case because it impairs the imputation mechanism. The interesting observation the difference between the pre- diction behavior of models when they have less information available. The model trained and tested on imputed data always reacts to inputs even in an unknown environment. While the predictions of (PROMISSING) models monotonically converge to 0.5, f.e. the models become indecisive in an unknown environment. As expected, the predictions of the PROMISSING model are slightly below the chance when the number of missing values increases, but this divergence is corrected in mPROMISSING. This feature is crucial in delicate applications of ML in the medical domain in which the decisions made by machines can significantly affect the quality of life of patients.
8

---

Under review as a conference paper at ICLR 2022
Label=0, Prediction=O.08
Lebel=1, Predictionn-0.99
Charge Sedetto
0.5
0.0
-0.5
-1.0
Label=1, Prediction00001
Label=O, Prediction=0.97
1.0
Change Fredzttonn
0.5
0.0
-0.5
Demograetii
Unoope
donato
Beetmeng
Aole
cns
Deroogmantia
Dligmaay
Comatio
Restmets
Hol
Groeioes
980
Removed Modality
Removed Modality
Figure 5: Counterfactual interpretation of model decisions using MPROMISSING for a) true nega- tive, b) true positive, c) false negative, and d) false positive examples in model predictions.
One possible application of PROMISSING is in counterfactual interpretation (CI) (Mothilal et al. 2020) of model decisions. CI is an effective method for the causal interpretation of complex mod- els such as NNs. PROMISSING facilitates the usage of CI; we can simply use artificially inserted missing values to create the counterfactual examples. In other words, to evaluate the effect of one variable or modality on the final prediction, we assume it is missing in a counterfactual scenario. To demonstrate this feature, we used this technique on the predictions of an mPROMISSING model. The first and second row of Fig. 5 show respectively the CI results for four example patients with correct and incorrect predictions on their remission. In the first case (Fig, 5la, the model correctly predicts no-remission, and CI shows that the demographic features are the decisive factor by re- ducing the probability of remission by 0.5. Similar inference can be used to explain the false decisions. For example, PANSS and MINI measures are decisive factors wrong model predic- tions in Fig. [5: and 55, respectively. This extra level of model explainability at the individual patient level is valuable in the clinical settings as answers the "Why?" question often asked by the clinicians about decisions of black box models, and hopefully, paves the way towards the new field of precision psychiatry (Fernandes et al., 2017).
SUMMARY AND CONCLUSIONS
In this work, we presented PROMISSING. a promising and novel second thought on dealing with missing data in neural networks. Unlike previous works, we do not intend to replace missing values based on their relation to other features or the joint distribution of observed data. Instead, we opt to use the unknowns as additional source of information by learning representation for missing data. The learned representation used to prune missing values by neutralizing their effect on the neurons' activation. The proposed method is convenient, intuitive, and reliable. It is convenient be- cause it is straightforward in implementation and application; there no need for extra data prepara- tion, and the missing values directly fed to the model; and unlike some data imputation methods, it is flexible with all input data types including continuous, binary, and categorical. This feature makes PROMISSING irreplaceable in applications of neural networks multi-modal datasets with a mixture of diverse features. is intuitive because reacts to unknowns like a living agent; the more the unknowns, the less the actions. Moreover, is reliable since it performs as well as al- ternatives while its decisions are calibrated with the amount of information it receives. Furthermore, provides an embedded mechanism explain the decisions of complex neural networks models which is favorable for more reliable applications of MI. in clinical decision making. Therefore, we consider PROMISSING as one step ahead towards more naturalistic ML-based decision-male in uncertain environments. Despite our extensive experiments, some analytical and empirical aspects of PROMISSING remain unexplored. Exploring the connection between modeling unknowns and model uncertainty, empirical comparison with model- basel incomplete data modeling, and applica- tions on more complex and structured data are among the unexplored avenues that we consider as future work.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of conclusions of this work, we have made the core implementation of the proposed method available in appendix A.3 The nanDense layer can be directly imported and tested by the readers. Furthermore, anonymized scripts and codes to reproduce the results for experiments on simulated data and OpenML datasets are included in the supplementary materials. Meanwhile, in case needed, we are open to sharing scripts of the experiments on clinical data with reviewers during the discussion process (the data is not available publicly). In our largest validation experiments in Sec. 3.2] we have used 52 publicly available ML datasets from OpenML. We tried best to be very specific, descriptive, and precise about the applied parameter settings, model specifications, and model evaluations in Sec. 3 Github upon acceptance of the manuscript.
We plan to release all the codes and scripts in
REFERENCES
Nancy C Annrreasen, William T Carpenter Jr, John M Kane, Robert A Lasser, Stephen R Marder, and Daniel R Weinberger. Remission in schizophrenia: proposed criteria and rationale for consensus. American Journal of Psychiatry, 162(3):441-449) 2005.
Melissa J Azur, Elizabeth A Stuart, Constantine Frangakis, and Philip J Leaf. Multiple imputation by chained equations: what is it and how does it work? International journal of methods in psychiatric research, 20(1):40-49, 2011.
Yoshua Bengio and Francois Gingras. Recurrent neural networks for missing or asynchronous data. Advances in neural information processing systems, pp. 395 9015401, 1996.
Cesar F Caiafa, Ziyao Wang, Jordi Sole-Casals, and Qibin Zhao. Learning from incomplete features by simultaneous training of neural networks and sparse coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2621-2630, 2021.
Francois Chollet et al. Keras, 2015. URL
Mark Collier, Alfredo Nazabal, and Christopher KI Williams. VAEs the presence missing data. arXiv preprint arXiv:2006.0530/ 2020.
Brisa S Fernandes, Leanne M Williams, Johann Steiner, Marion Leboyer, André F Carvalho, and Michael Berk. The new field of precision psychiatry. BMC medicine, 15,1):1-7,20 2017.
Pedro J Garcfa-Laanninna José-Luis Sancho-Gómez, and Aníbal R Figueiras- Vidal. Pattern classifi- cation with missing data: review. Neural Computing and Applications, 19(2):263-283 2010.
Zoubin Ghahramani and Michael I Jordan. Supervised learning from incomplete data via an em approach. In Advances in neural information processing systems. pp. 120-127, 1994.
Zoubin Ghahramani and Michael Jordan. Learning from incomplete data. 1995.
Huang, Anuj Pareek, Roham Zamanian, Imon Banerjee, and Matthew Lungren. Mul- timodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study pulmonary embolism detection. Scientific reports, 10(1):1-9, 2020.
Niels Bruun Ipsen, AlexanAee Mattei, and Jes Frellsen. not-MIWAE: Deep generative mod- elling with missing not at random data. In International Conference Learning Representations, 2020.
Sebastian Jäger, Arndt Allhorn, and Felix Bießmann. A benchmark for data imputation methods. Frontiers big Data, pp. 48, 2021.
Wei Jiang, Malgorzata Bogdan, Julie Josse, Szymon Majewski, Blažej Miasojedow, Veronika Ročková, and @raumaBase@) Group. Adaptive bayesian SLOPP: Model selection with incom- plete data. Journal of Computational and Graphical Statistics, (uuttaccceteed):1447, 2021.
Julie Josse, Nicolas Prost, Erwan Scomet, and Gaäl Varoquaux. On the consistency of supervised 2019.
learning with missing values. arXiv preprint
10

---

Under review as a conference paper at ICLR 2022
René S Kahn, Inge Winter van Rossum, Stefan Leucht, Philip McGuire, Shon W Lewis, Marion Leboyer, Celso Arango, Paola Dazzan, Richard Drake, Stephan Heres, et al. Amisulpride and olanzapine followed by open-label treatment with clozapine in first-piisode schizophrenia and schizophreniform disorder (optimise): a three-phase switching study. The Lancet Psychiatry, (10):797-807, 2018.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightghm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30:3146-3154 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980 2014.
Marine Le Morvan, Julie Josse, Thomas Moreau, Erwan Scornet, and Gaël Varoquaux. Neumiss networks: differential programming for supervised learning with missing values. In Advances in Neural Information Processing Systems 33, 2020.
Xuejun Liao, Hui Li, and Lawrence Carin. Quadratically gated mixture of experts for incomplete data classification. In Proceedings of the 24th International Conference Machine learning, pp- 553-560, 2007.
Roderick JA Little and Donald Rubin. Statistical analysis with missing data, volume 793. John Wiley & Sons, 2019.
Marine Le Morvan, Julie Josse, Erwan Scornet, and Gaël Varoquaux. What's a good imputation to predict with missing values? arXiv preprint arXiv:2106.000 00311, 2021.
Ramaravind K Mothilal, Amit Sharma, and Chenhao Tan. Explaining machine learning classifiers through diverse counterfactual explanations. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 607-617, 2020.
Alfredo Nazabal, Pablo M Olmos, Zoubin Ghahramani, and Isabel Valera. Handling incomplete heterogeneous data using VAEs. Pattern Recognition, 107:107501, 2020.
Robert K Nowicki, Rafal Scherer, and Leszek Rutkowski. Novel rough neural network for clas- sification with missing data. In 2016 21 International Conference on Methods and Models in Automation and Robotics (MMAR), pp. 820-825. IEEE, 2016.
Zdzislaw Pawlak. Rough theory and its applications data analysis. Cybernetics & &Systes, 29 (7):661-688, 1998.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathicu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-leam: Machine learning in python. Journal of machine learning research, 12(0ct):2825-2830, 2011.
Donald B Rubin. Inference and missing data. Biometrika, 63(3):581-592, 1976.
Joseph L Schafer and John W Graham. Missing data: our view of the state of the art. Psychological methods, 1(2):147, 2002.
Sebastian Schelter, Tammo Rukat, and Felix Biessmann. Jenga-a framework study the impact of data errors on the predictions of machine learning models. In EDBT, pp. 529-544, 2021.
Shaun R Seaman and lan R White. Review of inverse probability weighting for dealing with missing data. Statistical methods medical research, 22(3):278-22) 2013.
Marek Smieja, Lukasz Struski, Jacek Tabor, Bartosz Zieliński, and Przemyslaw Spurek. Processing of missing data by neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 2724-2734, 2018.
Alex J Smola, SVN Vishwanathan, and Thomas Hofmann. Kernel methods for missing variables. In International Workshop on Artificial Intelligence and Statistics, pp. 325-332, PMLR, 2005.
Bheki ETH Twala, MC Jones, and David J Hand. Good methods for coping with missing data in decision trees. Pattern Recognition Letters, 29(7):950-956 2008

---

Under review as a conference paper at ICLR 2022
Stef Van Buuren and Karin Groothuis- mice: Multivariate imputation by chained equa- tions in r. Journal of statistical software, 45(1):1-67, 2011
Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):99-60, 2014.
David Williams, Xuejun Liao, Ya Xue, and Lawrence Carin. Incomplete-data classification using logistic regression. In Proceedings of the 22nd International Conference on Machine learning, pp. 972 979, 2005.
Jeffrey Wooldridge. Inverse probability weighted estimation for general missing data problems. Journal of econometrics, 141(2):1281-130) 2007.
Jing Xia, Shengyu Zhang. Guolong Cai, Li Li, Qing Pan, Jing Yan, and Gangmin Ning. Adjusted weight voting algorithm for random forests in handling missing values. Pattern Recognition, 69: 52-60, 2017.
Jinsung Yoon, James Jordon, and Mihaela Schaar. Gain: Missing data imputation using generative adversarial nets. In International Conference on Machine Learning, pp. 5689-5698. PMLR, 2018.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 MISSING DATA MECHANISMS
Let X E <RXP represent the data matrix of n samples with features. Each sample æ E X then can be decomposed into observed x" and missing th components where the pattern of the missing values can be indicated by a mask vector m E {0, 1JP. Using these notation, the underlying mech- anisms for missing values in data can be divided into three categories (Rubin 1976 Ghahramani & Jordan 1995;
Schafer & Graham 2002):
1. Missing Completely at Random (MCAR): where there is no systematic relationship between the patterns of missingness with either observed or unobserved variables, i.e. p(m x", x²") = p(m). In this case the missingness patterns are completely at random and do not depend on observed or missing data.
2. Missing at Random (MAR): is a weaker assumption than MCAR (MCAR data is MAR but not vice-versa), in which the patterns of missingness are independent from missing data but may depend on observed values, i.e., p(m x x = p(m x²).
3. Missing Not at Random (MNAR): where the pattern of missingness depends on the values of missing data, i.e.p(mn p(m x", x)" depends on the values in æ. An example of MNAR is censored data in which a certain range of data is missing.
A.2 ACTIVATION OF (M)PROMISSING NEURONS
A.2.1 DERIVATION OF Eq. 3
Eq. can be derived by inserting Eq. 2 in Eq. for missing inputs zj E x"":
(kk) (8)
(x)
Σ
Σ
W + +(K))
(k)
5
I2W,
Ex
T
b(h) w (k)
(A)
Σ
(k)
+ ((())
Tilli
+
pmnj (k)
m
- (k)
+ (b)
(p r)k
Σ
T2W
(k)
+
Σ
+
p
p
E,620
Exx
+ gb(kk)
(k)
2 E
A.2.2 PROOF OF PROPOSITION 1
When all input values are missing (a:" = 2, q = 0. and r = p), then the activation of a PROMISS- ING neuron is zero:
pm (K)
(k))
Σ
(K)
+
Σ
x + ((t)
(k)
===
Em
B(0) (k)
(k)(k) (k)
a (k)
+ ((k)
+ (k)
r=p
(k) pro j
p
n. Cmm
a (k)
=0.
13

---

Under review as a conference paper at ICLR 2022
A.2.3 PROOF OF PROPOSITION
When there no missing values in inputs (x* 2, p, and 0), then the activation of normal neuron (Eq. is computed as follows:
(M)
Σ
Σ
2zm{h + b(4)
#:Ca
a (*)
T + ((K)
Ex
In this case, the activation of PROMISSING neuron is exactly equal to an ordinary neuron:
qblk)
a
Σ E.
(k)
Σ
(k) w,w; + (()).
p
A.2.4 PROOF PROPOSITION
When there are no missing values in inputs (a:" Ø, = and r 0), the activation of an MPROMISSING neuron is exactly equal to ordinary neuron:
gb(k) trwe (k)
(k))
Σ
(K)
(kk)
Σ 2/
x.wa ((k).
P
A.3 A NANDENSE LAYER
preliminary implementation of a nanDense layer listed in Listing D This layer is imple- mented by inheriting and overloading the class constructor, puild, and call functions. A flag, use_c, added to the class constructor to decide whether to use compensatory weights or not (defaulted to False). Using this flag, the user can simply switch between PROMISSING and mPROMISSING. Furthermore, the flag use_bias tra weight is concatenated to the end of the weight vector to implement compensatory weight in the case use c==True. The matrix of epsilons is also pre-set for usage in the call function. In the call function, the values for compensatory weights and neutralizers computed, accordingly. The current implementation is limited to non-sparse 2-dimensional inputs. The implementation is preliminary and optimized for memory and speed efficiencies. This implementation is tested with Python 3. 8 3. TensorFlow 2. 4 1, and Keras 2 3...3. 3. A similar guideline may be followed to implement PROMISSING within other types of NN layers.
fixed to True. the build function an ex-
import keras
import tensorflow as tf
from ccnsorflow.pythee framework import sparse_tensor
from censerflo.ppyther ops import gen_math_ops
from tensorflow.pyythn.co import math_ops
from tensorfiow.python.oss import nn_ops
From tenscrflow.ppttho..ek From fensorffov..yttoon.fraeeeeeee from Sensoriooov.yyhhoo.ttaaewweeer import tensor_shape
import backend as K
import dtypes
10 from Sensorrloow.pttton.ksrassse htt.::/www..oook........eeeeeeee www.eeeeeiineeeeennnennneeeeeeee spec import IpputSpec
11 from tenscriilo..prtooo.keeer. import activations
12 from tensorfloo..pytho..ker import initializers
from tensorflow.python
import tegularizers
14 from tenscrflow.python..r
import constraints
15
16
class nanDense (keras. Dense)
19
def init init__sellll (self,
20
units,
use_c False, # flag to use compensatory weight or
not.
activati on-None,
14

---

Under review as a conference paper at ICLR 2022
kernel huana___nntaallleeeooooooouu.ttd
24
biss_hittialll2e0==arrrr"" kernel_rogularrierroonn,
25
26
biss_regulari222::N0000
activitty.repullarieeee-nag
temme__costtrainn-lonnn
29
biss_constraintt-lnne
30
**kwargs);
super (nanDenss,, _ (units,
acdiity.eepuaciioraaiiiiyyy...ee reguaaaizz,,,,,.lliiieeee. **kwargs)
34
self.use_c use_c
35
self.use_bias True
36
self. = int (units) if not isinstance (anits, int) else units
self.actvvttion = activations.eet (activattion
38
eelikeernel_inttiallieer int.ialiiers..eeeeeeeee
39
solf.bbas_.inttialleee:: Intiaalleert.geeeetaaaaaaaiiiiit
40
s01f_Kerncl_coopuariie: = ccrulatteertsseet///eeoeeeouurur
solf.bias_rcgulaariees eggulaariberrssggttooooeeooaaaae
self.kermel_Gonttraint censtrainnts.ggetteeeeeeee henes
43
self.bias_.onstraant
44
self.input_spe: - InputSpec (min_ndim=2)
45
AAt.suuports_mmakiin True
47
def (sel,, input_shape).
48
49
ctype dtypes (self, aself.dtype dtype or K. f)oaat(())
if not (dtype. idlype.is_floating
dtype. iscomlee_cmmple
raise TypeError (" Unable build 'nanDense" layer with
floating point
52
dtyyp $s' $ (dtype,))
53
input_shape tensor_shape TensorShape (input_shape) last_dim censor_shapee dilaappeddmmmnioonneeee. value (input_shape [-1]) last_dim is None:
56
57
raise Valuekrror (he 1ast dimension of the inputs to
nanitense
sshulld defined. Found 'None'."
59
self Self.input_seee spec SpputSpec (min ndim=2, axes=(-1: last_dim))
60
if self.use_c $ extra weight if use_c is True.
self.kernel self weiight
63
'kernel', shape=[last_dimt1] sclf.units),
64
65
SouutarrtarrssllFeeeemLeeeaaaas constraintseeelfkkeemmcooooottt dtype=self.atyype, trainable-True)
68
69
70
else:
self. ferrne self aelf.add_weight
72.
'kernel',
73
shape-[last_ding self.units),
74
initiallieersall..eeeeeneiiiiii
regulariicerssel..eeeroo kernel_regularizeer
kenttaantsellleeeeee.oootot dtypo=self.dtypp, trainablo-Truc)
77
78
79
self self.add_weight.
'bias',
82
shape-
83
initiblleersslt.biaaa initialieee,
15

---

Under review as a conference paper at ICLR 2022
dtype=self dtype, trainable-True)
87
88
89
self.cpsilo = tf.fill woool..ernnl.... shape, x. k.opsilon()) ()) F Epsilon
Matrix
91
self.built True
92
93
def call (self, inputs):
96
dtype self.
97
if self. use weight to the weights
Computing and concatenating the compensatory
99
c tf.math. rww.ceeeeeeeeeee sum (tf. (tf.math. itantthii__annnnnniiiiii (inputs), dtype)
axis=1)
= maatthhiividd (c, inputs. shauts.shappll [1])
inputs tf.concat ([imputs, tf. exp.ptddddttxxxdeccccc.mmmm dims (c, axis=1) ), axis
102
103
kernel math_ops. add [self.epsil lon, sel f. f.renell Adding epsilon to weights
it seli.dtype:
106
if inputs. inpputs.dtyp..bur.. wtw.p.yypaaae...moee basepe.base_dyype !- dtype. dtype. www.bbbaseeeeeee
107
inputs - math_ops. cast (inputs, dtype-dtype)
rank inputsinti......... www.s.aapeaarr. rank
110
if rank or rank Nene: if isinstance (inputs, Searreettenorr.Sareeeeesssss raise NotimplementedError
111
clse:
outputs []
115
for i range (self.kernel. wwr.eerrlllsaae.
# Computing
Neutral izers and activati ons for each neuron
d tf. tf.math.divide (-self f.biasbaaassse [i] /selfslfff. f. kerkeernelll... shape
[0], kernel [:, i]) 117
temp inputs tf. where (tf.math. nan (inputs), d, replacing nans in inputs with -b/w
inputs)
kernel[; outputs. (gen math_aathpoooooo wwwpppp.......oeeoooo...oeoeeeep mul (temp inputs,
outputs tf. concat (outputs, axis=1)
120
121
$ Broadcast kernel to inputs.
else:
raise Not.Implementedrrre
125
outputs = molops.bias_eee add add (cutputs, self.bias)
if self. act activation vation is not None: outputs self self.actioo vatiiin (outputs)
129
130
return outputs
Listing I: An implementation of nanDense layer in Keras.
A.4 MISSING DATA SIMULATIONS
We used the same procedures in Schelter et al. 1222] to simulate the missing values in our exper- iments on simulated and OpenML data. For MAR and MNAR settings, first, a random range of certain feature F1 is selected. This random range is decided based random lower and upper bound percentiles. In the MAR condition, the values of a second random feature F2 are removed the values of F1 lie in the specified range, i.e., the missing values in F2 depend on the observations
16

---

Under review as a conference paper at ICLR 2022
Simulated Data
Zero Imputation
Mean Imputation
05
05-
10-
0.0
KNN Imputation
Iterative Imputation
0.
10
1.
-0
-05
(a) Simulated data and imputation results in MCAR setting.
Simulated Data
Zaro Imputation
Moan Imputation
20-
1.-
15-
0.0
0.-
1.
65
KNN Imputation
Iterative Imputation
1.5
10-
1.
-05
-05
0.5
(b) Simulated data and imputation results in MAR setting.
a) Simulated Data
b) Zero Imputation
Meen Impulation
1.0
0.5
d) KWN Imputation
c) Iterative Imputation
6.
-05-
10-
-05
(c) Simulated data and imputation results in MNAR setting.
Figure 6: each panel: a) The simulated XOR data. The effect of b) zero, e) mean, d) KNN, and e) iterative imputations on simulated data with MCAR, MAR, and MNAR missing values. The missing values for the first feature (x-axis) are selected based on a random range of the second feature (y-axis).
in F' (see Fig. fbr for an example). In contrast in the MNAR case, the values are removed from the same F1 feature. In fact, in the MNAR case the values of F1 censored for in certain range (see Fig. 6c for an example). In the MCAR setting, the samples with missing value are selected completely at random (see Fig. (a).
Fig. 6 illustrate the effect of applying different data imputation methods. In all cases, imputation may result in the misrepresentation of samples with missing values.
17

---

Under review as a conference paper at ICLR 2022
MCAR
MAR
MNAR
0.8
Zero
AUC
CAN
PRONSSONG
0.4
0.8
AUC
0.4
1000
Epochs
Epochs
Epochs
Figure 7: Comparison between learning curves of PROMISSING and MPROMISSING with data imputation approaches on simulated MCAR, MAR, and MNAR data, when tested on a test set without missing values (first row) and with 30% missing values (second row).
A.5 SUPPLEMENTARY MATERIAL FOR THE SIMULATION STUDY
To better understand the learned representation of unknowns U, we have visualized the elements in U across 100 simulation replications. Fig. S shows the values of across 100 simulation runs in our simulation study. In our simulation study, U has eight values (two inputs for each four input neurons). Markers with similar color and shape represent the neutralizer values for two features (axes) in four neurons in a specific run. For example, the four blue circles show the values of four neutralizers in wwo-dimensional feature space in the first run of the experiment. Since the scatter plot very crowded, we plotted only the 8 representative runs in Fig. in the main text in Sec. Since U is derived from the network parameters, due to the stochastic nature of parameter initialization and optimization, it may end up with different values in each run (local minima). Our visual inspection shows that the network finds four main groups of solutions for representing unknowns.
Fig. depicts elements in U for 8 selected runs that represent these four groups. In this case, U has eight elements (two inputs for four neurons). Markers with similar colors represent the neutralizer values for features (axes) and four neurons in specific run (legends). The neutralizers in the first group (1 and 11) are roughly zero imputers in which the missing values are imputed with close to (but not absolutely) zero values. These are sub-optimal solutions that happen only in a few runs. In the other cases, the models seem to perform more than simple imputer. In another minority group, represented by runs 52 and 100, the model turns to a mean imputer for the first feature, but it learns diverse representation for the second feature; the neurons see unknowns as borderline samples. In another minority group (represented by run 41 and 81), some neurons see unknowns as samples with extreme values (outside or close to the outer range of inputs). Run 7 and 10 represent the majority group of solutions which each neuron has its unique perception of unknowns.
A.6 SUPPLEMENTARY MATERIALS FOR EXPERIMENTS ON OPENML DATA
Table 2 and Table 3 summarize the OpenML datasets that respectively used in our classification
and regression analyses in Sec. 3.2 We used exactly the same datasets as in Jäger et al. (2211)
Fig. and Fig. 1ob compare the performance of our naive NN architecture (that are used our ex-
periments in Sec. 3.2 with random forest (RF) classifier or regressor across (with default settings) benchmark datasets. The errorbars represent the variation in AUCISMSE across 10 repetitions. The models are evaluated in 2-fold vross-validation scheme. The random number generator seed is fixed for all algorithms in each repetition. Our results show that our simple NN architecture with two hidden layers generally performs well as random forest models. Both NN and RF models show near the chance performance on 'pollen', egg-eye-state', and 'numerai28.6 thus we have removed these datasets from analyses in Sec. 3.2 About the regression tasks, again observed poor performance (SMSE> 1) of both models in the case of 'stock-fin fardamento02", 'wine-quality', Sharing-Demandd Demand datasets. Thus, they removed from analysis.
and "Bike- Mikke-Sharin--Dee
18

---

Under review as a conference paper at ICLR 2022
Fature
0.0
0.5
1.0
155
Feature 1
(a) The patterns of neutralizers for MCAR data.
1.5
Falure
0.5
-0.5
0.0
1.0
(b) The patterns of neutralizers for MAR data.
Fuluree2
(c) The patterns of neutralizers for MNAR data.
Figure 8: Patterns of neutralizer elements across two input features and the four neurons across 100 simulation repetitions. The markers with the same color and shape show the neutralizer values for each of four neurons in a specific run (the legends) of the pipeline. The blue and red dots in the background are representing the simulated data.
19

---

Under review as a conference paper at ICLR 2022
81
Feature 2
0.0
-0.5
0.0
15
Feature 1
Figure 9: Pattern of neutralizers in U across two input features and the four neurons. The markers with the same color show the neutralizer values for each four neurons in a specific run (legends) of the pipeline. The blue and red dots in the background are representing the simulated data.
Fig. II shows the results for experiments in Sec 3.2 in the regression tasks. Plots show the difference between the SMSE of the full model (trained on complete data) and the model trained on data with missing values. As expected the difference in SMSE increases with an increase in the ratios of missing samples and features. The (m)PROMISSING shows competitive performance compared to HGB and the better performing imputers, i.e., the zero, mean, and KNN imputers.
In an extra experiment, used the MCAR scheme to simulate different ratios of samples (0.1, 0.25, 0.5, 0.9) and features (0.1, 0.25, 0.5, 0.75, 1) with missing values across datasets. Please bear in mind that applying this setting (with more than one feature with missing values) to the MAR and MNAR conditions is tricky because guarantecing the assumptions behind these schemes can be complicated. Fig. 12 compares the performance of different data imputation methods with (M))ROMISSING the elassification tasks for different missing sample ratios (in each panel) and missing feature ratios (x-axes). The AUC drops with respect to the full model (trained on complete data) averaged across different datasets and ten runs. The (m)PROMISSING shows competitive performance compared to the better performing imputers, i.e., the mean and KNN imputers. Espe- cially, MPROMISSING performs slightly better when the ratio of missing features is increasing. A similar overall pattern can be observed when comparing the increase in the SMSE in the regression tasks (see Fig. 13). In summary, these results show that PROMISSING can compete with data imputation approaches without the need for filling the unknowns.
20

---

Under review as a conference paper at ICLR 2022
Classifier
Nib
0.0
atoonn
Emororm
houses
CIt predittion small
electriity
mu
Run Infermation
datuk Nanulk-ffoon coedharrd clens
MogCTwessog
tec
Artuan emplyye
Datasets
(a) AUCs in classification tasks. The higher score better.
Classifier
2.5
SASSE
0.5
StOCK Inddmeetoog
qualey
coles
BAGSeech aotthh
St.
miee Shartng Damand
deeaators
Zupbbres
COMET MC SAMPLE
(NCmm)
2
Dame mnt
Bretiinn
Datasets
(b) SMSEs in regression tasks. The less score is better.
Figure 10: Comparison between the performance of simple NN architecture with random forest across a) 31 classification and b) regression tasks.
21

---

Under review as a conference paper at ICLR 2022
Table 2: Classification datasets that are used in our analysis in Sec. 3.2
Dataset
OpenML ID
Sample Num.
Feature
Name
Num.
space-ga
737
3107
6
pollen
871
3848
5
wilt
40983
4839
5
analcatdata-supreme
728
4052
7
5
phoneme
1489
5404
5
delta-ailerons
803
7129
5
7
visualizing-soil
923
8641
4
bank8FM
725
8192
8
9
compas-two-years
42192
5278
13
bank-marketing
1558
4521
16
11
mammography
310
11183
6
mozilla4
1046
15545
5
13
wind
847
6574
14
14
churn
40701
5000
20
15
sylvine
41146
5124
20
16
ringnorm
1496
7400
20
17
twonorm
1507
7400
20
18
houses
823
20640
8
19
airlines
42493
26969
7
20
eeg-eye-state
1471
14980
14
21
MagicTelessoope
1120
19020
10
22
Amazon-enmploye--acess
4135
32769
9
23
BNG(tic-tac-too)
137
39366
9
24
BNG(breast-w)
251
39366
9
25
Click teec--redditioo-smeee
1220
39948
9
26
electricity
151
45312
8
27
fried
901
40768
10
28
881
40768
10
29
40922
88588
6
30
cofault-of-oreditt--a-------
42477
30000
23
31
numerai28.6
23517
96320
21
MCAR
MAR
MNAR
0.20
0.20
0.15
240
0.15
0.15
0.10
SMIE
0.05
0.05
0.10
0.25
0.50
0.90
0.10
0.25
0.50
0.90
0.10
0.25
0.50
0.90
Semple Missing Rate
Sample Missing Rate
Sample Missing Rate
Figure 11: Comparing increase in SMSEs in regression tasks with respect to the full model across five different imputation methods, HGB, and (M)RROMISSING in MCAR, MAR, and MNAR set- tings.
22

---

Under review as a conference paper at ICLR 2022
Table 3: Regression datasets that are used in our analysis in Sec. 3.2
Dataset
OpenML ID
Sample Num.
Feature
Name
Num.
foockffaddmeent022
42545
6277
auml-eml-l-
42675
4585
10
delta-clevators
198
9517
sulfur
23515
10081
kin8nm
189
8192
8
wine-quality
287
6497
12
Long
42636
4477
19
Brazilian-houses
42688
10692
12
dataset-sales
42183
10738
14
10
BoG(echoMonths)
1199
17496
11
cpu-act
197
8192
21
12
house-8L
218
22784
8
13
Sikk-Sharing-Deaeee
42712
17379
BNG(Lowbwt)
1193
31104
15
elevators
216
16599
18
16
2dplanes
215
40768
10
17
COMET-MC-SAMPLE
23395
89640
diamonds
42225
53940
19
BNG(stock)
1200
59049
20
BNG(mv)
1213
78732
10
21
auml-url-2
42669
95911
12
Sample Missing Rate-0.10
Sample Missing Rate-0.25
0.08
AUC
0.02
0.01
0.25
0.50
1.00
0.25
0.75
Feanure Mixsing Rate
Feature Mixing Rate
Sample Missing Rate=0.50
Sample
0.3
Zoro
0.15
ALIC. Drop
0.2
0.10
MFROMISINWG
0.0
0.10
0.25
0.50
0.75
1.00
0.10
0.25
0.50
0.75
1.00
Feature
Feature Missing
Figure 12: Comparing AUC drops with respect to the full model across four different imputation methods, HGB, and (m)PROSISSING in classification tasks.
23

---

Under review as a conference paper at ICLR 2022
Sample Missing Rate-D.10
Sample Missing Rate-0.
0.10
0.2
SMSE
0.05
0.1
0.25
Fesshrw Mivsing Rate
Festtre Moving Rate
Sample Missing Rate=0.55
Sample Missing Rate=0.90
SMSE Incresse
0.4
0.6
0.3
0.4
0.2
0.2
0.1
0.10
0.25
0.50
0.75
1.00
0.10
0.25
0.50
0.75
1.00
Feature
Feature
Figure 13: Comparing increase in SMSEs with respect to the full model across four different impu- tation methods, HGB, and (M)PROMISSING in regression tasks.
24


Reviews:
[{"review_id": "ZMsw_aBQ50u", "reviewer": "Reviewer_vNRV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Approaches that can readily for data with missing values are crucially important.\n\nHowever, the approach here is based on handwaving, with intuitions that seem fragile. It does not empirically perform better than mean imputation.\n\nThe introduction claims \"it has been shown that constant imputation is only effective when the missing features are not informative (Josse 2019)\". This is not at all what Josse 2019 shows.\n\nPage 3 claims that the approach enables relaxing the MAR assumption, but there is no legit argument to this claim.\n\nThe goal of neuralizing the effect of missing values on activations inside the architecture is not a desirable one. Indeed, suppose that we have two features, X1 abs X2, and y only depends on X2, but X1 and X2 are correlated. Should X2 be missing, an architecture attempting to make a prediction solely from the observed data and using the same logic add the fully-observed case, would fall, given that the fully-observed case relies only on X2. Rather, am adequate architecture would then rely on capturing and using the link between X1 and X2, and could achieve good predictions.\n\nFigure 2 should show other approaches, such as mean imputation.", "summary_of_the_paper": "This submission contributes an approach to handle missing values in Neural networks by replacing inside the architecture the missing values by placeholders which cancel the role of the feature in the architecture. The approach is benchmarked empirically, but does not appear to outperform mean imputation.", "summary_of_the_review": "The contribution is based on intuitions that do not seem very solid and should be better studied. It does not really perform better than mean imputation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AZp5HgLHi-z", "reviewer": "Reviewer_FmEA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points:\n- The paper is well-motivated and the method is interesting.\n- I appreciate that the authors have included a NN architecture implementing the proposed approach for use with a popular NN modeling software.\n- The data analyses are fairly thorough, but could be improved (see below).\n\nMajor concerns:\n- There is a large literature on weighting methods (e.g., inverse probability weighting, doubly-robust estimators) that was ignored in this paper. You should include some references to this literature and discuss how your proposed method is related (or not) to these methods.\n- What is the motivation behind the 'neutralizer' value? How should we interpret this?\n- I'm not convinced that keeping unknowns as unknowns allows the MAR assumption to be relaxed (page 3). If there truly is data missing-not-at-random, then it seems *any* approach suffers from this, regardless of whether or not the missing mechanism is modeled (which is a difficult or impossible task with MNAR data).\n- The take-home message from the simulations and data analyses, in my view, is that PROMISSING does no worse than methods that use imputation. This is at odds with the motivation, which describes potential for increased performance over these other methods. Why do we not see improvements in the experiments and analyses?\n- In the simulations:\n    - what does performance look like if the data are MAR? or MNAR?\n    - how does PROMISSING perform with no missing data (i.e., compare PROMISSING/mPROMISSING trained without missing values to the NN trained without missing values -- are they the same?)?\n    - why is there such a large decrease in predictive performance when the test set has missing values vs when it doesn't (Figure 1)?\n    - 50 Monte-Carlo replications is quite small (section 3.1); consider increasing to at least 500\n    - I find it difficult to digest Figure 3, and I found the discussion on page 6 unsatisfying. Consider re-framing this paragraph or removing the figure.\n- In the data analyses:\n    - again, why did you focus on MCAR? did you consider MAR or MNAR?\n    - again, 10 replications is too small, run at least 100\n    - Figure 4 is difficult to digest, because the lines are too close together. What are the magnitudes of the difference in performance between approaches? What is the Monte-Carlo standard error? I suspect that the differences between methods are quite small relative to the error.\n\nMinor concerns:\n- You could make more clear that equation (1) is the traditional approach to NN activation, while your approach augments this. At first, it is confusing to say that you need imputation to fit (1) when in reality you fit a different objective.\n- Check for typos and language use throughout.\n", "summary_of_the_paper": "In this paper, the authors propose a method titled PROMISSING; this provides a new approach to handling missing data. Rather than imputation, a complete-case analysis, or inverse probability weighting, among other methods, the authors advocate for learning a problem-specific numerical representation for unknowns. The approach is interesting, and the experiments and data analyses are a good start at understanding the method.", "summary_of_the_review": "I vote for accepting the paper subject to some additional experiments, based on my review above.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4bRToIZGHmv", "reviewer": "Reviewer_DWUQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strong points\n- The computationally and conceptually light approach to handle missing values in neural networks appears to be an interesting alternative to overly simple imputation (such as mean or 0 imputation) and more costly imputation strategies (iterative imputation, multiple imputation). The empirical findings suggest that it performs at least as good as more complex imputation strategies, at least in the MCAR case, which is encouraging.\n- An important and application-oriented aspect of this approach is that it seems to reflect the uncertainty of the missing values in the predictions (e.g., the probabilities in the case of binary classification in their clinical example get closer to 0.5 as the amount of missingness increases). This is an interesting feature and I would encourage an analytical/theoretical assessment of this aspect in a revised version or future work in this direction.\n- The experiments and results of Section 3 are well presented and commented and the details provided on the method and the simulations allow for easy reproducibility of the results.\n- The article is well written and provides an adequately succinct yet sufficient bibliography of missing values handling in statistics and machine learning.\n\n## Issues/Points that require clarification\n- From the introduction and the description of the experiments it does not become clear to me which case this method is most suited for. To my understanding, PROMISSING should be applicable in cases where classical (conditional) imputation would also work, i.e., when the data is MAR (or maybe also MNAR?) and the outcome does not depend on the missingness pattern itself. In this case the outcome $y$ is defined via the complete data and not the incomplete data. This could maybe be added explicitly to distinguish this paper's setting from the case of potentially predictive missingness patterns (Le Morvan et al., 2021).\n- The patterns of neutralizers presented in Figure 3 suggest that certain missing values are detected as \"predictive\" of the outcome (runs 7, 9, 3 and 5). It would be interesting to rerun the experiments and plot a similar figure but in the case of MNAR data where the missingness pattern can indeed be predictive (to some extent) of the outcome. This point relates to the previous comment about the exact setting, especially the data generating process, that is considered here.\n- Choice of methods in Section 3.2: Since the authors perform a quick experiment on complete data to compare their architecture's performance to the results from random forest classifiers and regressors, why don't they also compare their method to random forests with missing values handling that does not require imputation (such as missing incorporated in attributes (Twala et al., 2008) that is implemented in the module sklearn.ensemble of scikit-learn or that can be implemented manually as suggested [here](https://rmisstastic.netlify.app/how-to/python/predict_html/how\\%20to\\%20predict))?\n- Almost a minor comment: what is the impact of data standardization in the OpenML Data section (3.2), especially for the proposed method(s)?\n\n### Minor comments (that did not impact the score)\n- p.1: I would add that removing incomplete samples is even impossible in high-dimensional settings and not only an issue in small-sample size datasets.\n- p.2: Some works exist in high-dimensional settings with missing values such as Jiang et al. (2021).\n- p.2: anything in the rest of network $>>$ anything in the rest of _the_ network\n- p.5: converge _slower_ $>>$ converge _more slowly_\n- p.9: This feature is _clinical_ $>>$ This feature is _critical_?\n- p.9: _several analytical and empirical_ aspects of PROMISSING _remains_ unexplored $>>$ _analytical and several empirical_ aspects of PROMISSING _remain_ unexplored (I didn't find any analytical aspects of PROMISSING discussed in the paper)\n\n######################################\n### Post-rebuttal update\nI thank the authors for their detailed and timely responses. The proposed additional experiments on MAR and MNAR data support the authors' claims. I followed all exchanges between the authors and the other reviewers who criticize the lack of contributions/novelty of the proposed method. I agree to a certain extent with the other reviewers, especially concerning the justification and derivation of the proposed neutralizer as well as its theoretical grounding. However, I would acknowledge the authors' motivation to provide a simple-to-use method that allows to adapt the prediction indecisiveness to the level of available information which is in important feature, especially in medical contexts. In their extensive simulation study and real-world application they demonstrate the performance of this method and its comparison to other existing methods without claiming to perform better but being conceptually easier to integrate in existing architectures. \nTherefore I will increase my score and recommend accepting this paper. In case of acceptance (or for a re-submission at another venue in case of rejection), I strongly encourage the authors to modify the introduction and the conclusion of their paper for the camera-ready to emphasize the current lack of theoretical understanding of this approach (currently they only write \"some analytical and empirical aspects\nof PROMISSING remain unexplored\"). Pointing out this current lack of theoretical understanding won't hurt the contribution of this empirical work but could encourage other researchers to take up on these open questions and to try to theoretically understand the empirically observed behavior.\n\n######################################\n\n### References\n[1] Wei Jiang, Małgorzata Bogdan, Julie Josse, Szymon Majewski, Błażej Miasojedow, Veronika Ročková, and TraumaBase® Group. Adaptive Bayesian SLOPE: Model Selection with Incomplete Data. _Journal of Computational and Graphical Statistics_, 2021.\n\n[2] Marine Le Morvan, Julie Josse, Thomas Moreau, Erwan Scornet, and Gaël Varoquaux. NeuMiss networks: differentiable programming for supervised learning with missing values. _arXiv preprint arXiv:2007.01627_, 2020.\n\n[3] Marine Le Morvan, Julie Josse, Erwan Scornet, and Gaël Varoquaux. What's a good imputation to predict with missing values?. _arXiv preprint arXiv:2106.00311_, 2021.\n\n[4] Bheki ETH Twala, M. C. Jones, and David J. Hand. Good methods for coping with missing data in decision trees. _Pattern Recognition Letters_, 29(7), 2008.", "summary_of_the_paper": "The present paper proposes an alternative to imputation or list-wise deletion in the context of neural networks and incomplete features. Missing values are replaced by a data-specific numerical representation that is learned at the same time as the rest of the network. The handling of the missing values is located in the neurons of the first layer and each missing value is \"replaced\" by a neuron-specific neutralizer in its activation function. They empirically show that their approach is comparable to several imputation techniques. In a clinical application they show that their model becomes indecisive as the amount of missingness (in terms of missing features) increases, which is a potentially interesting feature for clinical prediction models.", "summary_of_the_review": "In summary, the proposal of this paper is interesting and promising, but it lacks clarification of the problem setting and justifications w.r.t. the choice of the representation of the missing values as well as the positioning of this approach w.r.t. other methods such as NeuMiss (Le Morvan et al., 2020). Importantly as well, a more extensive simulation study would be helpful to assess the behavior of the proposed strategy, especially looking at other missingness mechanisms than MCAR alone. I would encourage the authors to at least either add more justifications and/or results in the analytical part or to extend their simulation study to a wider range of possible settings to allow for an appropriate assessment of their proposal.\nI will read the rebuttal carefully and am willing to increase the score if the authors address the raised concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "33SpNTYou5A", "reviewer": "Reviewer_Bfyk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The manuscript presents a novel approach to dealing with missing values, the architecture they propose is a modification on the fully connected layer which allows the user to plug this layer into any neural network. This makes this method particularly versatile since it can have many uses beyond the ones presented in the manuscript. The methodology of the paper is well written and the methodology is mathematically sound. Experiments and results sections could use an improvement to enhance clarity given the heterogeneity of analyses conducted. That being said, there are some comments that I think should be addressed:\nMajor comments:\n- The baselines you compared your method to contain only a few state-of-the-art methods. I would add the iterative imputer with ridge regression, and add a few of the newer methods that are well performing. I would suggest GAIN (Yoon 2018, last in the references) and GRAPE (You et al. 2020 arXiv:2010.16418) as state-of-the-art baselines.  In addition, you could also compare to an input where the missing value flags are inserted as an input to the NN which is an approach that is widely used that also forgoes imputation. This would give the paper more strength especially to make sure that your model outperforms others in the conditions mentioned.\n- The results in figure 2 show overlapping error bars. First, it is important to specify the error bar meaning (in other figures as well) and also cannot claim that the PROMISSING method surpassed the kNN method given the overlapping error bars unless you have performed an appropriate statistical test.\n- It is unclear to me why different imputation methods are being utilized in different experiments (kNN is used only for the psychosis dataset while 4 methods are utilized for simulated data) and also different result visualization methods (AUC vs AUC drop vs model trajectory).  I understand that you are trying to make a different point each time but it is natural to discuss the general performance on the psychosis task before the trajectory. One possibility is that the imputation is able to generate better predictions even in the absence of a large number of modalities. \n- Additionally, it appears from figure 6 that there is a higher number of patients with positive output than with negative output which could be the reason why the regular network is showing the positive bias when the number of missing modalities is high. While there are mitigation strategies for that in regular neural networks, this could also be a strength point for your model. It is important to report the ratio of imbalance as it would explain why the baseline model ends up with a bias as you remove more features and then you can discuss how your model mitigates this.\n\nMinor comments:\n- There is an assertion the methods that PROMISSING the features are not inferred from observed features causing the MAR assumption to be relaxed. This is not entirely accurate given that the training phase will be affected by the missing values so it will still work best when the data is MAR similar to how regression-based imputation methods perform best at those conditions. Also, having the data MNAR makes the problem of inference easier but less generalizable to patterns of MNAR data of patterns never seen before. I would suggest removing this part and replacing by a statement about how this method forgoes imputation steps which can be costly in the case of complex imputation mechanisms.\n- In section 3.1, it is mentioned that the RandomForest Regressor is equivalent to MICE, does that mean you used multiple imputation (data augmentation) or used a single imputation variant? Also citation of the paper for this method is missing (https://doi.org/10.1093/bioinformatics/btr597)\n- For the OpenML dataset, it is not mentioned what data imputation method is utilized with the RF benchmark. In addition, in table 2, there are some datasets that have zero features, is that a typo?\n- There are many missing technical details in the testing procedures that I would suggest to be added to the supplementary info such as the training/test split ratios, hyperparameter-selection procedure (if exists), number of training epochs... etc. This information is crucial for reproducibility.\n- Regarding figure 3, the figure even after simplification is quite difficult to comprehend because of the missing distinction between the 8 different weights. I suggest choosing one neuron and plotting the 2 weights that feed into that (or even one) this would decrease a lot of clutter. Also maybe it is good to plot the original weights without missingness to visualize the effect of missingness.\n- I am unsure of how (un)balanced the datasets used here are. It is beneficial to report those values because for highly unbalanced datasets (quite common in medical data), other measures such as precision and recall are much more valuable and representative of the performance.\n- In figure 4, the MCAR scheme used to artificially remove values is unclear to me how it was achieved. I am guessing that feature missing rate meant the maximum number of features that had missing values and sample missing rate is the amount of missing values that are missing in a given sample. I would first like to confirm if that were true. Additionally, there are some points that need clarification here: 1/ Is that sample missing rate calculated as a percentage of the feature missing rate? 2/ Are the features removed constant across samples or do you randomize that choice over samples as well?\n- Second question above can also be posed at the modalities removed in figure 6.\n", "summary_of_the_paper": "The paper provides a novel method of handling missing values in neural network architectures by replacing the missing input with a pruning signal that causes the network to utilize the existing values only. The author(s) tested the method on simulated data as well as real data on a group of classification and regression tasks.", "summary_of_the_review": "The method presented in this paper is novel and presents an interesting handling of the missing data problem. It can even be tested on DNN-based imputation such as generative models. Given the versatility of the model, it has so much potential for many uses. The main weakness of the model is the lack of significant improvement in performance in comparison to the baseline models but this should also be commended as the author(s) did not cherry pick the results where the model is well-performing and rather opted to report all their results. The other weakness is the small number of baseline models the new method is compared to. I believe the authors should summarize where their method shines vs where it does not in the discussion or summary because the way the method is pitched in the current format is that it leads to better performance than the imputation-based methods in the basic tasks of classification/regression while the results do not reflect that. Additionally more baseline methods should be added to the comparison to enable proper comparison.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2KcJ69CGED0", "reviewer": "Reviewer_eatM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is a very simple approach to handling missing data, and is best understood in eq 3 -- basically the bias b^{(k)} to hidden unit j has been downweighted by q/p (the fraction of observed to the total number of inputs). No justification is given as to why this transformation makes sense. (For example, one might think that one should *upweight* term \\sum_{x_ in x^o} x_i w^{(k)}_i by a factor of p/q to \"compensate\" for the missing inputs.)\n\nThe paper does not do a good job of correctly reviewing the literature on missing data. It is unforgivable not to have a proper discussion of MCAR, MAR and MNAR in the main text (not a footnote), and to correctly point out that this classification is due to Rubin (1976) [not the reference to Azur et al, 2011].\n\nIt fails to describe the concept of multiple imputation, i.e to make use the *distribution* p(x^m|x^o) when producing predictions as p(y|x^o) = \\int p(y|x^o,x^m) p(x^m|x^o) dx^m . This should nicely induce greater uncertainty as more missing data is encountered.\n\nOne other standard approach to prediction problems with missing data is to make use of the concept of the \"indicator\" vector m, which takes on values of 1 or 0 to indicate if an input is missing or present resp. See e.g the reference to NeuMiss networks for an example of this, although it has been in use for many years.  In fact one can propose a more general version of what the authors propose by having the concatenation (x,m) of length 2p as input to the network.  When m_i = 1, then x_i takes on a value of 0. If m_i = 0, then x_i takes on its observed value. One has weights to the k-th hidden unit from both x_i and m_i. The current proposal is obtained by having a weight of value -b^{(k)}/p from m_i. This value is shared over all inputs in the PROMISSING proposal, but in fact it would be more natural to learn the parameters of the m_i inputs unconstrained.\n\nThe paper then conducts 3 sets of experiments, on simulated XOR-type data, OpenML data, and a clinical application.\n\nThe XOR data is a simple mixture of 4 Gaussians dataset, but is perhaps a poor choice for a missing data experiment, as with only two input variables either 0 or 50% of the inputs are missing. BTW it is possible to work out analytically the best performance that can be obtained on such as dataset, using a MoG model as in Ghahramani and Jordan (1995). While both PROMISSING methods do as well as can be expected on this dataset (Fig 2b), we really should also be given results for competitor methods (zero imputation, mean imputation, KNN imputation, MICE imputation etc).\n\nFor the OpenML datasets, the given results are averaged over different datasets.  It would be most natural to consider the differences between different methods on *each* dataset, and then produce some aggregate summary. It is also essential to consider if the differences between methods are *statistically significant* -- here the 10-times repetition of experiments should help with this assessment.  See e.g.  T. Dietterich, Neural Computation 10 1895-1923 (1998) for more details on this. I note from Fig 4 that there seems to be very little difference between the PROMISSING methods and the very standard mean imputation.\n\n\nResults on the clinical PPP task give AUC values on a par with the KNN results. (BTW, in answer to the text opposite Fig 5, one can use *mode* rather than *mean* imputation for categorical variables -- this is standard.)\n\nIt is also worth noting that there has been quite a lot of work about NNs for handling missing data in VAEs; this is relevant\twork to\tthis paper.  See e.g. the HI-VAE of Nazabal et al (2020,\nhttps://arxiv.org/abs/1807.03653), which simply \"zeros out\" missing inputs, \"VAEs in the Presence of Missing Data\" by Collier et al (2020, https://arxiv.org/pdf/2006.05301.pdf) and not-MIWAE (Ipsen et al,\n2020, https://arxiv.org/abs/2006.12871). These last two papers use indicator\tvariable method.\n\nOverall, this paper proposes the (m)PROMISSING methods for handling missing data in neural networks. The proposal is in fact a simplified version of the well-known indicator variable method for handling missing data, as shown above. The experiments carried out do not show any significant advantage over standard methods of data imputation.  And the synthetic experiments are carried out in the easiest MCAR case, while MNAR is of most practical interest. For the reasons above I see no case to accept this paper.\n\nOther points:\n\nsec 2 -- I recommend p^o and p^m instead of q and r -- it is very hard to remember what p, q and r mean.\n\nsec 3.3.1 -- rebalancing the data to 50/50 is poor practice -- with an imbalance of only 67:33, I would not carry out any adjustment.  With a classifier trained on a different class ratio than used at test time, one should adjust the classifier post-training, as described e.g. in Bishop, Neural Networks for Pattern Recognition (1995) p 223.\n\nLittle and Rubin book -- you cite the 3rd edition, but note the first was in 1987.", "summary_of_the_paper": "This paper tackles the problem of missing data by proposing the replacement of the input to the k-th hidden neuron in the first hidden layer from the j-th input as in eq 2. Experiments are conducted comparing this so-called \"(m)PROMISSING\" method with other methods for treating missing data.", "summary_of_the_review": "Overall, this paper proposes the (m)PROMISSING methods for handling missing data in neural networks. The proposal is in fact a simplified version of the well-known indicator variable method for handling missing data, as shown above. The experiments carried out do not show any significant advantage over standard methods of data imputation.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a simple and intuitive method to prune the missing value in the learning and inference steps of the neural networks, leading to similar prediction performance as other methods to impute missing value. It has some really useful insights, but could benefit from one more round of revision for a strong publication: 
1. improving the writing so that its sets up the right expectations on the contributions of the paper; 
2. providing discussions on its connections (and differences) with zero-imputation and missing-indicator methods; 
3. thoroughly investigating the experiment results to illustrate the advantages of the proposed method.  

The recommendation of reject is made based on the technical aspect of the paper.
-----------------------------
During the rebuttal phase, the authors misused the interactive and transparent (for the better or worse) openreview system by writing inappropriate comments with personal accusations to the reviewers who write negative reviews. We would like to extend the apologies to the reviewers for this unpleasant experience and thank the reviewers for their engagement and work, as well as their fair assessment of the paper.
Decision: Reject