Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This submission contributes an approach to handle missing values in Neural networks by replacing inside the architecture the missing values by placeholders which cancel the role of the feature in the architecture. The approach is benchmarked empirically, but does not appear to outperform mean imputation.
main_review: Approaches that can readily for data with missing values are crucially important.

However, the approach here is based on handwaving, with intuitions that seem fragile. It does not empirically perform better than mean imputation.

The introduction claims "it has been shown that constant imputation is only effective when the missing features are not informative (Josse 2019)". This is not at all what Josse 2019 shows.

Page 3 claims that the approach enables relaxing the MAR assumption, but there is no legit argument to this claim.

The goal of neuralizing the effect of missing values on activations inside the architecture is not a desirable one. Indeed, suppose that we have two features, X1 abs X2, and y only depends on X2, but X1 and X2 are correlated. Should X2 be missing, an architecture attempting to make a prediction solely from the observed data and using the same logic add the fully-observed case, would fall, given that the fully-observed case relies only on X2. Rather, am adequate architecture would then rely on capturing and using the link between X1 and X2, and could achieve good predictions.

Figure 2 should show other approaches, such as mean imputation.
summary_of_the_review: The contribution is based on intuitions that do not seem very solid and should be better studied. It does not really perform better than mean imputation.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The present paper proposes an alternative to imputation or list-wise deletion in the context of neural networks and incomplete features. Missing values are replaced by a data-specific numerical representation that is learned at the same time as the rest of the network. The handling of the missing values is located in the neurons of the first layer and each missing value is "replaced" by a neuron-specific neutralizer in its activation function. They empirically show that their approach is comparable to several imputation techniques. In a clinical application they show that their model becomes indecisive as the amount of missingness (in terms of missing features) increases, which is a potentially interesting feature for clinical prediction models.
main_review: ## Strong points
- The computationally and conceptually light approach to handle missing values in neural networks appears to be an interesting alternative to overly simple imputation (such as mean or 0 imputation) and more costly imputation strategies (iterative imputation, multiple imputation). The empirical findings suggest that it performs at least as good as more complex imputation strategies, at least in the MCAR case, which is encouraging.
- An important and application-oriented aspect of this approach is that it seems to reflect the uncertainty of the missing values in the predictions (e.g., the probabilities in the case of binary classification in their clinical example get closer to 0.5 as the amount of missingness increases). This is an interesting feature and I would encourage an analytical/theoretical assessment of this aspect in a revised version or future work in this direction.
- The experiments and results of Section 3 are well presented and commented and the details provided on the method and the simulations allow for easy reproducibility of the results.
- The article is well written and provides an adequately succinct yet sufficient bibliography of missing values handling in statistics and machine learning.

## Issues/Points that require clarification
- From the introduction and the description of the experiments it does not become clear to me which case this method is most suited for. To my understanding, PROMISSING should be applicable in cases where classical (conditional) imputation would also work, i.e., when the data is MAR (or maybe also MNAR?) and the outcome does not depend on the missingness pattern itself. In this case the outcome $y$ is defined via the complete data and not the incomplete data. This could maybe be added explicitly to distinguish this paper's setting from the case of potentially predictive missingness patterns (Le Morvan et al., 2021).
- The patterns of neutralizers presented in Figure 3 suggest that certain missing values are detected as "predictive" of the outcome (runs 7, 9, 3 and 5). It would be interesting to rerun the experiments and plot a similar figure but in the case of MNAR data where the missingness pattern can indeed be predictive (to some extent) of the outcome. This point relates to the previous comment about the exact setting, especially the data generating process, that is considered here.
- Choice of methods in Section 3.2: Since the authors perform a quick experiment on complete data to compare their architecture's performance to the results from random forest classifiers and regressors, why don't they also compare their method to random forests with missing values handling that does not require imputation (such as missing incorporated in attributes (Twala et al., 2008) that is implemented in the module sklearn.ensemble of scikit-learn or that can be implemented manually as suggested [here](https://rmisstastic.netlify.app/how-to/python/predict_html/how\%20to\%20predict))?
- Almost a minor comment: what is the impact of data standardization in the OpenML Data section (3.2), especially for the proposed method(s)?

### Minor comments (that did not impact the score)
- p.1: I would add that removing incomplete samples is even impossible in high-dimensional settings and not only an issue in small-sample size datasets.
- p.2: Some works exist in high-dimensional settings with missing values such as Jiang et al. (2021).
- p.2: anything in the rest of network $>>$ anything in the rest of _the_ network
- p.5: converge _slower_ $>>$ converge _more slowly_
- p.9: This feature is _clinical_ $>>$ This feature is _critical_?
- p.9: _several analytical and empirical_ aspects of PROMISSING _remains_ unexplored $>>$ _analytical and several empirical_ aspects of PROMISSING _remain_ unexplored (I didn't find any analytical aspects of PROMISSING discussed in the paper)

######################################
### Post-rebuttal update
I thank the authors for their detailed and timely responses. The proposed additional experiments on MAR and MNAR data support the authors' claims. I followed all exchanges between the authors and the other reviewers who criticize the lack of contributions/novelty of the proposed method. I agree to a certain extent with the other reviewers, especially concerning the justification and derivation of the proposed neutralizer as well as its theoretical grounding. However, I would acknowledge the authors' motivation to provide a simple-to-use method that allows to adapt the prediction indecisiveness to the level of available information which is in important feature, especially in medical contexts. In their extensive simulation study and real-world application they demonstrate the performance of this method and its comparison to other existing methods without claiming to perform better but being conceptually easier to integrate in existing architectures. 
Therefore I will increase my score and recommend accepting this paper. In case of acceptance (or for a re-submission at another venue in case of rejection), I strongly encourage the authors to modify the introduction and the conclusion of their paper for the camera-ready to emphasize the current lack of theoretical understanding of this approach (currently they only write "some analytical and empirical aspects
of PROMISSING remain unexplored"). Pointing out this current lack of theoretical understanding won't hurt the contribution of this empirical work but could encourage other researchers to take up on these open questions and to try to theoretically understand the empirically observed behavior.

######################################

### References
[1] Wei Jiang, Małgorzata Bogdan, Julie Josse, Szymon Majewski, Błażej Miasojedow, Veronika Ročková, and TraumaBase® Group. Adaptive Bayesian SLOPE: Model Selection with Incomplete Data. _Journal of Computational and Graphical Statistics_, 2021.

[2] Marine Le Morvan, Julie Josse, Thomas Moreau, Erwan Scornet, and Gaël Varoquaux. NeuMiss networks: differentiable programming for supervised learning with missing values. _arXiv preprint arXiv:2007.01627_, 2020.

[3] Marine Le Morvan, Julie Josse, Erwan Scornet, and Gaël Varoquaux. What's a good imputation to predict with missing values?. _arXiv preprint arXiv:2106.00311_, 2021.

[4] Bheki ETH Twala, M. C. Jones, and David J. Hand. Good methods for coping with missing data in decision trees. _Pattern Recognition Letters_, 29(7), 2008.
summary_of_the_review: In summary, the proposal of this paper is interesting and promising, but it lacks clarification of the problem setting and justifications w.r.t. the choice of the representation of the missing values as well as the positioning of this approach w.r.t. other methods such as NeuMiss (Le Morvan et al., 2020). Importantly as well, a more extensive simulation study would be helpful to assess the behavior of the proposed strategy, especially looking at other missingness mechanisms than MCAR alone. I would encourage the authors to at least either add more justifications and/or results in the analytical part or to extend their simulation study to a wider range of possible settings to allow for an appropriate assessment of their proposal.
I will read the rebuttal carefully and am willing to increase the score if the authors address the raised concerns.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a method titled PROMISSING; this provides a new approach to handling missing data. Rather than imputation, a complete-case analysis, or inverse probability weighting, among other methods, the authors advocate for learning a problem-specific numerical representation for unknowns. The approach is interesting, and the experiments and data analyses are a good start at understanding the method.
main_review: Strong points:
- The paper is well-motivated and the method is interesting.
- I appreciate that the authors have included a NN architecture implementing the proposed approach for use with a popular NN modeling software.
- The data analyses are fairly thorough, but could be improved (see below).

Major concerns:
- There is a large literature on weighting methods (e.g., inverse probability weighting, doubly-robust estimators) that was ignored in this paper. You should include some references to this literature and discuss how your proposed method is related (or not) to these methods.
- What is the motivation behind the 'neutralizer' value? How should we interpret this?
- I'm not convinced that keeping unknowns as unknowns allows the MAR assumption to be relaxed (page 3). If there truly is data missing-not-at-random, then it seems *any* approach suffers from this, regardless of whether or not the missing mechanism is modeled (which is a difficult or impossible task with MNAR data).
- The take-home message from the simulations and data analyses, in my view, is that PROMISSING does no worse than methods that use imputation. This is at odds with the motivation, which describes potential for increased performance over these other methods. Why do we not see improvements in the experiments and analyses?
- In the simulations:
    - what does performance look like if the data are MAR? or MNAR?
    - how does PROMISSING perform with no missing data (i.e., compare PROMISSING/mPROMISSING trained without missing values to the NN trained without missing values -- are they the same?)?
    - why is there such a large decrease in predictive performance when the test set has missing values vs when it doesn't (Figure 1)?
    - 50 Monte-Carlo replications is quite small (section 3.1); consider increasing to at least 500
    - I find it difficult to digest Figure 3, and I found the discussion on page 6 unsatisfying. Consider re-framing this paragraph or removing the figure.
- In the data analyses:
    - again, why did you focus on MCAR? did you consider MAR or MNAR?
    - again, 10 replications is too small, run at least 100
    - Figure 4 is difficult to digest, because the lines are too close together. What are the magnitudes of the difference in performance between approaches? What is the Monte-Carlo standard error? I suspect that the differences between methods are quite small relative to the error.

Minor concerns:
- You could make more clear that equation (1) is the traditional approach to NN activation, while your approach augments this. At first, it is confusing to say that you need imputation to fit (1) when in reality you fit a different objective.
- Check for typos and language use throughout.

summary_of_the_review: I vote for accepting the paper subject to some additional experiments, based on my review above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper provides a novel method of handling missing values in neural network architectures by replacing the missing input with a pruning signal that causes the network to utilize the existing values only. The author(s) tested the method on simulated data as well as real data on a group of classification and regression tasks.
main_review: The manuscript presents a novel approach to dealing with missing values, the architecture they propose is a modification on the fully connected layer which allows the user to plug this layer into any neural network. This makes this method particularly versatile since it can have many uses beyond the ones presented in the manuscript. The methodology of the paper is well written and the methodology is mathematically sound. Experiments and results sections could use an improvement to enhance clarity given the heterogeneity of analyses conducted. That being said, there are some comments that I think should be addressed:
Major comments:
- The baselines you compared your method to contain only a few state-of-the-art methods. I would add the iterative imputer with ridge regression, and add a few of the newer methods that are well performing. I would suggest GAIN (Yoon 2018, last in the references) and GRAPE (You et al. 2020 arXiv:2010.16418) as state-of-the-art baselines.  In addition, you could also compare to an input where the missing value flags are inserted as an input to the NN which is an approach that is widely used that also forgoes imputation. This would give the paper more strength especially to make sure that your model outperforms others in the conditions mentioned.
- The results in figure 2 show overlapping error bars. First, it is important to specify the error bar meaning (in other figures as well) and also cannot claim that the PROMISSING method surpassed the kNN method given the overlapping error bars unless you have performed an appropriate statistical test.
- It is unclear to me why different imputation methods are being utilized in different experiments (kNN is used only for the psychosis dataset while 4 methods are utilized for simulated data) and also different result visualization methods (AUC vs AUC drop vs model trajectory).  I understand that you are trying to make a different point each time but it is natural to discuss the general performance on the psychosis task before the trajectory. One possibility is that the imputation is able to generate better predictions even in the absence of a large number of modalities. 
- Additionally, it appears from figure 6 that there is a higher number of patients with positive output than with negative output which could be the reason why the regular network is showing the positive bias when the number of missing modalities is high. While there are mitigation strategies for that in regular neural networks, this could also be a strength point for your model. It is important to report the ratio of imbalance as it would explain why the baseline model ends up with a bias as you remove more features and then you can discuss how your model mitigates this.

Minor comments:
- There is an assertion the methods that PROMISSING the features are not inferred from observed features causing the MAR assumption to be relaxed. This is not entirely accurate given that the training phase will be affected by the missing values so it will still work best when the data is MAR similar to how regression-based imputation methods perform best at those conditions. Also, having the data MNAR makes the problem of inference easier but less generalizable to patterns of MNAR data of patterns never seen before. I would suggest removing this part and replacing by a statement about how this method forgoes imputation steps which can be costly in the case of complex imputation mechanisms.
- In section 3.1, it is mentioned that the RandomForest Regressor is equivalent to MICE, does that mean you used multiple imputation (data augmentation) or used a single imputation variant? Also citation of the paper for this method is missing (https://doi.org/10.1093/bioinformatics/btr597)
- For the OpenML dataset, it is not mentioned what data imputation method is utilized with the RF benchmark. In addition, in table 2, there are some datasets that have zero features, is that a typo?
- There are many missing technical details in the testing procedures that I would suggest to be added to the supplementary info such as the training/test split ratios, hyperparameter-selection procedure (if exists), number of training epochs... etc. This information is crucial for reproducibility.
- Regarding figure 3, the figure even after simplification is quite difficult to comprehend because of the missing distinction between the 8 different weights. I suggest choosing one neuron and plotting the 2 weights that feed into that (or even one) this would decrease a lot of clutter. Also maybe it is good to plot the original weights without missingness to visualize the effect of missingness.
- I am unsure of how (un)balanced the datasets used here are. It is beneficial to report those values because for highly unbalanced datasets (quite common in medical data), other measures such as precision and recall are much more valuable and representative of the performance.
- In figure 4, the MCAR scheme used to artificially remove values is unclear to me how it was achieved. I am guessing that feature missing rate meant the maximum number of features that had missing values and sample missing rate is the amount of missing values that are missing in a given sample. I would first like to confirm if that were true. Additionally, there are some points that need clarification here: 1/ Is that sample missing rate calculated as a percentage of the feature missing rate? 2/ Are the features removed constant across samples or do you randomize that choice over samples as well?
- Second question above can also be posed at the modalities removed in figure 6.

summary_of_the_review: The method presented in this paper is novel and presents an interesting handling of the missing data problem. It can even be tested on DNN-based imputation such as generative models. Given the versatility of the model, it has so much potential for many uses. The main weakness of the model is the lack of significant improvement in performance in comparison to the baseline models but this should also be commended as the author(s) did not cherry pick the results where the model is well-performing and rather opted to report all their results. The other weakness is the small number of baseline models the new method is compared to. I believe the authors should summarize where their method shines vs where it does not in the discussion or summary because the way the method is pitched in the current format is that it leads to better performance than the imputation-based methods in the basic tasks of classification/regression while the results do not reflect that. Additionally more baseline methods should be added to the comparison to enable proper comparison.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper tackles the problem of missing data by proposing the replacement of the input to the k-th hidden neuron in the first hidden layer from the j-th input as in eq 2. Experiments are conducted comparing this so-called "(m)PROMISSING" method with other methods for treating missing data.
main_review: The proposed method is a very simple approach to handling missing data, and is best understood in eq 3 -- basically the bias b^{(k)} to hidden unit j has been downweighted by q/p (the fraction of observed to the total number of inputs). No justification is given as to why this transformation makes sense. (For example, one might think that one should *upweight* term \sum_{x_ in x^o} x_i w^{(k)}_i by a factor of p/q to "compensate" for the missing inputs.)

The paper does not do a good job of correctly reviewing the literature on missing data. It is unforgivable not to have a proper discussion of MCAR, MAR and MNAR in the main text (not a footnote), and to correctly point out that this classification is due to Rubin (1976) [not the reference to Azur et al, 2011].

It fails to describe the concept of multiple imputation, i.e to make use the *distribution* p(x^m|x^o) when producing predictions as p(y|x^o) = \int p(y|x^o,x^m) p(x^m|x^o) dx^m . This should nicely induce greater uncertainty as more missing data is encountered.

One other standard approach to prediction problems with missing data is to make use of the concept of the "indicator" vector m, which takes on values of 1 or 0 to indicate if an input is missing or present resp. See e.g the reference to NeuMiss networks for an example of this, although it has been in use for many years.  In fact one can propose a more general version of what the authors propose by having the concatenation (x,m) of length 2p as input to the network.  When m_i = 1, then x_i takes on a value of 0. If m_i = 0, then x_i takes on its observed value. One has weights to the k-th hidden unit from both x_i and m_i. The current proposal is obtained by having a weight of value -b^{(k)}/p from m_i. This value is shared over all inputs in the PROMISSING proposal, but in fact it would be more natural to learn the parameters of the m_i inputs unconstrained.

The paper then conducts 3 sets of experiments, on simulated XOR-type data, OpenML data, and a clinical application.

The XOR data is a simple mixture of 4 Gaussians dataset, but is perhaps a poor choice for a missing data experiment, as with only two input variables either 0 or 50% of the inputs are missing. BTW it is possible to work out analytically the best performance that can be obtained on such as dataset, using a MoG model as in Ghahramani and Jordan (1995). While both PROMISSING methods do as well as can be expected on this dataset (Fig 2b), we really should also be given results for competitor methods (zero imputation, mean imputation, KNN imputation, MICE imputation etc).

For the OpenML datasets, the given results are averaged over different datasets.  It would be most natural to consider the differences between different methods on *each* dataset, and then produce some aggregate summary. It is also essential to consider if the differences between methods are *statistically significant* -- here the 10-times repetition of experiments should help with this assessment.  See e.g.  T. Dietterich, Neural Computation 10 1895-1923 (1998) for more details on this. I note from Fig 4 that there seems to be very little difference between the PROMISSING methods and the very standard mean imputation.


Results on the clinical PPP task give AUC values on a par with the KNN results. (BTW, in answer to the text opposite Fig 5, one can use *mode* rather than *mean* imputation for categorical variables -- this is standard.)

It is also worth noting that there has been quite a lot of work about NNs for handling missing data in VAEs; this is relevant	work to	this paper.  See e.g. the HI-VAE of Nazabal et al (2020,
https://arxiv.org/abs/1807.03653), which simply "zeros out" missing inputs, "VAEs in the Presence of Missing Data" by Collier et al (2020, https://arxiv.org/pdf/2006.05301.pdf) and not-MIWAE (Ipsen et al,
2020, https://arxiv.org/abs/2006.12871). These last two papers use indicator	variable method.

Overall, this paper proposes the (m)PROMISSING methods for handling missing data in neural networks. The proposal is in fact a simplified version of the well-known indicator variable method for handling missing data, as shown above. The experiments carried out do not show any significant advantage over standard methods of data imputation.  And the synthetic experiments are carried out in the easiest MCAR case, while MNAR is of most practical interest. For the reasons above I see no case to accept this paper.

Other points:

sec 2 -- I recommend p^o and p^m instead of q and r -- it is very hard to remember what p, q and r mean.

sec 3.3.1 -- rebalancing the data to 50/50 is poor practice -- with an imbalance of only 67:33, I would not carry out any adjustment.  With a classifier trained on a different class ratio than used at test time, one should adjust the classifier post-training, as described e.g. in Bishop, Neural Networks for Pattern Recognition (1995) p 223.

Little and Rubin book -- you cite the 3rd edition, but note the first was in 1987.
summary_of_the_review: Overall, this paper proposes the (m)PROMISSING methods for handling missing data in neural networks. The proposal is in fact a simplified version of the well-known indicator variable method for handling missing data, as shown above. The experiments carried out do not show any significant advantage over standard methods of data imputation.
review_title: null

Meta-review: The paper presents a simple and intuitive method to prune the missing value in the learning and inference steps of the neural networks, leading to similar prediction performance as other methods to impute missing value. It has some really useful insights, but could benefit from one more round of revision for a strong publication: 
1. improving the writing so that its sets up the right expectations on the contributions of the paper; 
2. providing discussions on its connections (and differences) with zero-imputation and missing-indicator methods; 
3. thoroughly investigating the experiment results to illustrate the advantages of the proposed method.  

The recommendation of reject is made based on the technical aspect of the paper.
-----------------------------
During the rebuttal phase, the authors misused the interactive and transparent (for the better or worse) openreview system by writing inappropriate comments with personal accusations to the reviewers who write negative reviews. We would like to extend the apologies to the reviewers for this unpleasant experience and thank the reviewers for their engagement and work, as well as their fair assessment of the paper.
Decision: Reject