Knockout: A simple way to handle missing inputs
Authors: Minh Nguyen
Batuhan K. Karaman
Heejong Kim
Alan Q. Wang
Fengbei Liu
Mert R. Sabuncu
Venue: Accepted by TMLR ()

Abstract: Deep learning models benefit from rich (e.g., multi-modal) input features. However, multimodal models might be challenging to deploy, because some inputs may be missing at inference. Current popular solutions include marginalization, imputation, and training multiple models. Marginalization achieves calibrated predictions, but it is computationally expensive and only feasible for low dimensional inputs. Imputation may result in inaccurate predictions, particularly when high-dimensional data, such as images, are missing. Training multiple models, where each model is designed to handle different subsets of inputs, can work well but requires prior knowledge of missing input patterns. Furthermore, training and retaining multiple models can be costly. We propose an efficient method to learn both the conditional distribution using full inputs and the marginal distributions. Our method, Knockout, randomly replaces input features with appropriate placeholder values during training. We provide a theoretical justification for Knockout and show that it can be interpreted as an implicit marginalization strategy. We evaluate Knockout across a wide range of simulations and real-world datasets and show that it offers strong empirical performance.

Published in Transactions on Machine Learning Research (07/2025)
Knockout: A simple way to handle missing inputs
Minh Nguyen!
bn24/@cormelleedv
Cornell University
Batuhan K. Karaman
kbk46&ccornell.co edu
Cornell University
Heejong Kim Weill Cornell Medicine
hek4004 @emed.cornelll edu
Alan Q. Wang Cornell University
aw847Beornell. edu
Fengbei Liut Cornell University
B4558coorellLedu
Mert R. Sabuncu
msabuccu@ccrnnell.ce
Cornell University and Weill Cornell Medicine for the Alzheimer's Disease Neuroimaging Initiative"
Reviewed on OpenReview: https: openreview. net/ ptr/form? id= K71y5pge88
Abstract
Deep learning models benefit from rich (e.g,, multi-modal) input features. However, multi- modal models might be challenging to deploy, because some inputs may be missing at inference. Current popular solutions include marginalization, imputation, and training mul- tiple models. Marginalization achieves calibrated predictions, but it is computationally expensive and only feasible for low dimensional inpuls. Imputation may result in inaccurate predictions, particularly when high-dimensional data, such as images, are missing. Training multiple models, where each model designed handle different subsets of inputs, can work well but requires prior knowledge of missing input patterns. Furthermore, training and retaining multiple models can be costly. We propose efficient method to learn both the conditional distribution using full inputs and the marginal distributions. Our method, Knockout, randomly replaces input features with appropriate placeholder values during training. We provide theoretical justification for Knockout and show that can be inter- preted as an implicit marginalization strategy. We evaluate Knockout across a wide range of simulations and roal-world datascts and show that it offers strong empirical performance.
1 Introduction
In many real-world applications of machine learning (ML. and statistics, not all variables might be available for every data point. This issue, also known missingness, is well-studied in the literature (Little & Rubin 2019) and common in fields like healthcare, social sciences, and environmental studies. From Bayesian perspective, missingness can be solved by marginalization, where we would like model to marginalize out
Data used in preparation of this article were obtained from the Alzheimer's Discase Neuroimaging Initiative (ADNT) database lodii.oomiuse usc.edu). As such, the investigatons within the ADNI contributed to the design and implementation of ADNI and/or provided data but did participate in analysis writing of this report. A complete listing of ADNI investigators can be found http://adni.oon.uscc. wwwddd//ww-ooeetttt. /uploads///oooo how to__pply/ADDi ADNT Acknowledgement, List.pdf f; Equal contribution

---

Published in Transactions on Machine Learning Research (07/2025)
the missing variables from the conditioning set. However, during training, often do not know which features will be missing at inference.
In lieu of training multiple models for every missingness pattern, a common strategy is imputation, which uses a point estimate (usually the mean or mode or a constant) to impute the missing features (Le Morvan et al. 2020bj. This can be seen as approximating the marginalization with a delta function. More sophisticated imputation methods use EM (Josse et al., 2019) or neural-based networks QMattei & Frellsen| 2019 [psen et al. 2022). Although many prior methods may work well in some instances, they may not scale readily to nigh-dimensional inputs like images (Kyono et al. 2021 You et al. 2020), require additional networks for generation of missing variables {[psen et al. (022). , only apply to continuous inputs {Le Morvan et al.
2020a; 2021). assume linearity of predictors (Le Morvan et al. 2020b), or make assumptions about the data distribution HHzaan et al. 2015).
In this work, we propose a simple, effective, and theoretically-justffii augmentation strategy, called Knock- out, for handling missing inputs. During (raining, features are randomly "knocked out" and replaced by constant "placeholder" values. At inference time, using the placeholder value corresponds mathematically to estimation with the appropriate marginal distribution. Knockout can be seen as implicitly maximizing the likelihood of a weighted sum of the conditional estimators and all desired marginals in a single model. We demonstrate the broad applicability of Knockout in a suite of experiments using synthetic and real-world data from multiple modalities mamaage-bssed and tabular). Real world experiments include Alzheimer's forecasting, noisy label learning, multi-modal MR image segmentation multi-modal image classification, and multi-view tree genus classification. We show the effectiveness of Knockout in handling low and high- iigh-dimensional missing inputs compared against competitive baselines.
2
Method
2.1
Background
The goal of supervised ML is to learn the conditional distribution p(Y|X) where Y is the output (pre- dictive target) and X E EN is the vector of inputs or features. The prediction for a new sample æ is i arg maxy p(Y|X z). However, in many practical applications, not all features may be present. When feature X; is missing, the vector X_; denotes the non-missing features. In general, multiple features may be missing at time. We can represent this with a missingness indicator set M and corresponding non-missing features as X-M. In this case, what we really want is p(Y X -)). How can we account for missingness?
A simple approach is to train a separate model for p( M)) ie. a model that takes only the non-missing features X-M as inputs. However, this is expensive because separate model needed for each missingness pattern. Furthermore, there is no sharing of information between these separate models, even though they are theoretically related.
Another approach rewrites - using the already available x(Y|X):
(1(X-M) - / p(Y, XM X dM = P( XX)
(1)
The goal now is to obtain ((MMXX methods approximate Eq. ① by replacing P(MM|X-M with delta function. For example, "mean impu- (ation" uses the mean of the missing features XM, i.e. M], for XM itself. In Eq. [ this corresponds to approximating X(MM|XXX R ((B()M)), a delta function. While convenient and commonly used, mean imputation ignores the dependency between XM and X-M. and does not account for any uncertainty.
and perform the integration over all possible XM- Imputation
More sophisticated imputation methods capture the interdependencies between inputs (Troyanskaya et al. 2001 Stekhoven & Bühlmann| 2012), for example by explicitly modeling p(MM|X-M) by training a separate model. The point estimate Z = arg max x,, p( x(XM|X_MM) can be used at inference time for the missing XM- While properly accounting for interdependencies between inputs, this approach requires fitting a separate model for ( In multiple imputation, multiple samples from pl X M X -M) are drawn
2

---

Published in Transactions on Machine Learning Research (07 (07/2025)
and Monte Carlo approximation is used to estimate the integral the RHS of Eq. 1 Mattte & Frellsen, 2019 Kyono et al. 2021 [psen et al. 2022). Although this is more accurate than single imputation, it is not effective in high dimensional space.
2.2 Knockout
We propose simple augmentation strategy for neural network training called Knockout that enables estima- tion of the conditional distribution pXY] and all desired marginals nonlinear model, such as a deep neural network. During training, features are augmented by randomly "knocking out" and replacing them with constant, "placcholder" values. At inference time, using the place- holder value corresponds mathematicalyy to estimation with the suitable marginal distribution.
in single, high capacity,
Specifically, let M M2,,, MN] E {0, 1 denote a binary, induced missingness indicator vector, Let क [T1, 22.......... ÆN] E RN denote vector of placcholder values. Then, define ('M, M o @ (1 M) @ X as augmented Knockout inpuls, where 1 is a vector of ones and @ denoles element-wise multiplication. During one training iteration, a different Knockout imput is used corresponding to different randomly sampled M for every data sample. The model weights are trained to minimize the loss function with respect to Y, as is done regularly.
Two mild conditions are required ensure proper training. First, the placeholder values must be "appro- priate," we will elaborate below. For our theoretical treatment, we will out-of support values appropriate; i.c. IM & Support(XM). Second, M must be independent of X and Y, Lee M LL X, X,Y It follows straightforwardly that these two conditions lead to modeling the desired conditional and marginal distributions simultaneously. First, since EM is not in the support of XM,
X'M IM + MM=1 11
X'M + EM + MM =0 0 and X'M M -XM-
(2)
where 0 and 1 are vectors of zeros and ones of appropriate shape. Second, since M is independent of X and Y, it follows that imputing with the default value ΣM is equivalent to marginalization of the missing variables defined by M:
X' M MM1MM=11M MMMMMM))
(3)
-
(4)
At the two extremes, no Knockout (M = 0) corresponds to the original conditional distribution, and full Knockout (M = 1) to the full marginal:
x' -
(5)
x)) = - pYY).
(6)
For a new test input æ, the prediction when xM missing is simply
arg max X arg arg X X' M-)
(7)
i.e., the learned estimator with the augmented Knockout input.
2.2.1 Knockout as an Implicit Multi-task Objective
The missingness indicator M determines how inpuls are replaced with appropriate placeholder values dur- ing training. To satisfy the independence condition of M with X and Y, the variables M are sampled independently from a distribution p(M) during training. We show that this training strategy can be viewed as a multi-task objective (Caruana 1997 decomposed as a weighted sum of terms, where each term is a separate marginal weighted by the distribution of M. Let denote the loss function be minimized (e.g..
mean-squared-error cross-entropy loss):
Note is necessary that M,, My for any 1,j-
3

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
L(0) = Ex,, v E(Y; (X X))
(8)
= Ex,y Σ I(M-m) (X
(9)
mEM
= -xxy Σ
pMM=m) ((Y;Jo(X'(m, X)))
(10)
mEM
Σ p(M=m)
(11)
where I is the indicator function.
If there knowledge about the missingness patterns at inference (e.g,, some X; and X, exhibit correlated missingness), one can design p(M) appropriately to cover all the expected missing patterns, i.e. by sampling m during training with different weights. In the absence of such knowledge, the most general distribution for M is i.i.d. Bernoulli. A common way correlated missingness arises in real-world applications is in structured inputs like latent features or images, where the entire feature vector whole image is missing. In our experiments, we demonstrate the superiority of structured Knockout, over naive i.i.d. Knockout, when such correlated missingness is known a priori.
2.3 Choosing Appropriate Placeholder Values
Our thcoretical analysis assumes that the placcholder value I; lies outside the support of X; (sec Ap- pendix A.2 for further discussion). Previous work by Josse et al. (2019) and Bertsimas et al. (0024) has shown that, with infinite data, mean imputation (for continuous variables) and out-of-support imputation (for both continuous and discrete variables) can be optimal. However, in practice, data is finite, and em- pirical performance becomes more important. such cases, using out-of-support values as placeholders is often preferable and can be mathematically justified partleppattrllllyy when X, is low-dimensional. That said, for high-dimensional inputs such vectors images, out-of-range placcholders may lead to practical issucs like unstable gradients or limited model capacity, making them less effective. In the sections that follow, we relax the out-of-support assumption and offer practical recommendations for choosing placeholder values for different types of X,, based on these considerations.
2.3.1 Non-structured
In this section, recommend suitable placeholder values for non-structured, scalar-valued inputs.
Categorical variable: T; can be Nx, + if X; are integer-valued classes from to Nx,- one-hot encoded, I; can be a vector of Os.
Continuous variable and Non-empty Infeasible Set: we can scale X, to [0. and choose T; = 1. More generally, if X, has unbounded range but a non-empty infcasible set, then H; can be set to value in the infeasible set. For example, if X; only takes positive values, then we can set 24 -1.
Continuous variable and Empty Infeasible Set: we suggest applying Z-score normalization and choosing T; such that it lies in low probability region of the normalized X; such that p(X;=7,) ss 0. As we argue in Appendix A.1 this approach leads to an approximation the desired marginal.
Although different distributions have different regions of low probability, we use the behavior of the Gaussian distribution as a guide to choose h.. Fig. showws the histogram of the norm of points sampled from standard Gaussian distributions with different dimensionality. For a univariate standard Gaussian, most of the points lic close to the origin so we should choose 3, far away from the origin. However, as the dimension increases, most of the points lie on the hyper-sphere away from the origin so we should choose F, to be the point origin (i.e. vector of zeros). Table summarizes the choices of 5; for different types of inputs X;-
4

---

Published in Transactions on Machine Learning Research (07 (025)
Dimension =
Dimension =
Dimension = 16
Dimension = 64
Probability
0.02
0.00
5 Nxrm(X)
10
0
5
10
0
5
10
0
5
10
NXrm(X)
Nxrm(X)
Norm(X)
Figure 1: The region of high density of standard Gaussian shifts away from the origin as the number of dimensions increases. This motivates different choices of placeholder values at different dimensions.
Table 1: List of different types of X; and the recommended 2;
Type of X;
Example
Dimension
Support
Normalized?
Categorical
Gender
1
{l, [x..
N/A
Nx. +1
Continuous
Test scores
1
[a
Scale to [0, 1]
-1
Contimous
Temperature
1
[a, 0o) or (-o0, b]
Scale to [0, 00)
-1
Continuous
White noise
1
(00,,00)
7-score
+10
Structured Structured
Images
1000
[a, bj 00)
Scale to [0, 1]
0
Latent vectors
>1
Z-score
0
2.3.2 Structured
For structured inputs like images and feature vectors, there are multiple choices of placeholder. However, we found that Knockout applied with some out-of-support placeholder like -1, cause issues like unstable gradients. Therefore, recommend appropriate placeholder to be either the image of all Os the mean image. When Z-score normalization is applied, the image and the mean image coincide. Theoretically, it is well known that the mean of high dimensional random variable, such as a Gaussian, has very low probability [Vershynin] 2018) (also see Fig. []. We believe this recommendation balances the tension be- tween ensuring an extremely-low probability placeholder with proper convergence and performance. For an empirical demonstration, see Appendix A.3
2.4 Observed Missingness during Training
The treatment above assumes complete training data, and inference-time missingness only. We now consider the situation where training data has observed missing inputs. Let N be the binary mask indicating the ob- served data missingness. N is different from M, which denotes the missingness induced by Knockout during training. Thus, N is fixed for data sample, while M is stochastic. Observed missingness generally falls under the following scenarios (Little & Rubin 2019).
Missing Completely at Random (MCAR): This implics that N JL X.Y. Let M' = N M be the augmented masking indicator, where V denotes the logical OR operation. Since N IL X, Y and M Y,Y, obtain the same result in Section 2.2 when using M' instead of M
M' IL X,Y. Therefore, the masking indicator vector. This implics that Knockout can be applied to MCAR training data simply by masking all the missing values using the same placeholders T.
Missing at Random (MAR) and Missing not at Random (MNAR): This implies that N J X,Y. Thus, we cannot replace the missing values training data using the same placcholders. However, we can substitute these values using placeholders that are different from & but are also outside the support of the input variables (or very unlikely values). Tet the placeholders for the data missingness be + * T. During
5

---

Published in Transactions on Machine Learning Research (07/2025)
Table 2: Summary of experimental setups
Task
Type of X;
Dimension
Normalized?
Simmlations
Categorical Mategorica/Coontinoaa Conntinuus
Z-score (Cont. X,)
-10
Alzheimer's Forecasting
Continuous
1
Z-score
10
-10
Privileged Information
Continuous
Scale [0. 1]
N/A
Tumor Segmentation
Structured (images)
256
Scale to [0, 1]
0
N/A
Tree Genus Classification
Structured (latent)
768 or 2048
Z-score
N/A
Prostate Cancer Detection
Structured (latent)
256
Z-score
N/A
Food Classification
Structured (latent)
768
Z-score
N/A
training, Knockout still randomly masks out input variables, including those that are not observed in the data. Thus, the results in Section 2.2 still hold since M 1L X.Y.
During inference, if we know priori that 2; of a sample is missing not at random, then we can use è; as the placeholder. Otherwise, if we know x; is missing completely at random, we use Ty. Unless stated otherwise, Knockout always uses a different placeholder for MAR and MNAR variables.
3 Related Work
Knockout is inspired by other methods with different objectives. Dropout (Srivastava et al. 2014 Gal & Ghahramani 2016) prevents overfitting by randomly dropping units (hidden and visible) during training, effectively marginalizing over model paramcters. During inforence, this marginalization can be approxi- mated by predicting once without dropout (Srivastava et. al., 2014) or averaging multiple predictions with dropout (Gal & Ghahramani 2016p. Blankout (Maaten et al. 2013) and mDAE (Chen et al., 2014) learn to marginalize out the effects of corruption over inputs. In contrast, Knockout learns different marginals to handle varied missing imput patterns.
Imputation techniques impute missing inputs explicitly, e.g., using mean, median, or mode values. Model- based imputation methods predict missing values via k-nearest neighbors (Troyanskaya et al. 2001), chained equations (Van Buuren & Groothuis Ouddshorn 2011), random forests (Stekhoven & Bühhmann. 2012)
autoencoders (Gondara & Wang. 2018 Ivanov et al.] 2019: Tall & Robinson 2022). ,, RNN (Nguyen et al.
2020
GANs (Yoon et al.) 2018 Li et al. 2019 Belghazi et al.) 2019), normalizing flows (Li ct al, Although more accurate than simple imputation, model-based imputation incurs significant additional
2020
computation costs, especially with high- sigh-dimensional inputs. Some approaches (Ma et al. 2021 Peis et al. require additional training of multiple VAEs or sub-networks. Other approaches (Mattci & Frellson
2022
2019
Ma et al. 2019) require training only one VAE but assune homogeneous data (all continuous variables or all binary variables), limiting flexibility. In contrast, Knockout uses a single classifier, avoiding the need for separate models to impute missing inputs explicitly.
Another relevant line of work is causal discovery (Spirtes et al. 2000 which often involves fitting model
using different subsets of available inputs and multiple distributions simultaneously (Lippe et al.) 2022
James et al. )023). To reduce computational cost, it is common to train a single model that can handle
different subsets of inputs using dropout (Brouillard et al.) 2020 Lippe et al. 2022 Ke et al. 2023
Nguyen
& Sabuncu 2024 Nguyen et al. 2024a).
Techniques like Knockout are often used in practice train a single neural network that models multiple distributions, but are often justified empirically with little care taken in choosing placeholder values. Many
works use zeros without theoretical justification (Belghazi et al.) 2019 Ke et al.) 2023 Brouillard et al. 2020 Lippe et al. )022). GAIN (Yoon et al. 2018) and MisGAN (Li ct al. 2019) impute using out-of-support values similar to Knockout. However, both are limited in their Treatment by assuming that the supports are bounded, and do not consider categorical variables. While the approach similar to some prior work for structural inputs (Neverova al.] 2015 Parthasarathy Sundaram, 2020) or low-dimensional inputs (Bert- simas et al. 2024). Knockout's theoretical backing shows that can handle multiple data types and multiple missingness types (emplet//MCAA//MA//MM)
Similarly, methods like Selective MTM (Van Ness et al.
6

---

Published in Transactions on Machine Learning Research (07 (07/202) /025)
a) Complete
b) MCAR
c) MNAR (self-censored)
0
10
10
10
10
10
1
10
-1
10
10"
0
2
3
0
2
3
2
3
mFr KNNNN
Knock
ZI
MIWAE
Knock
Drop
-0-LI
CB
MICE
sMIWAE
CB
Figure 2: Test MSE evaluated against Bayes optimal prediction () averaged over 10 repetitions (lower is better). X axis indicates the number of missing variables at inference time. mFr: missForest, Drop: dropout, Knock: Knockout, CB: common baseline, ZI: zero-impiin sero-imputation with mask, sMIWAE: supMIWAE. KNN: k-ncarest-neighbor imputation, LI: linear regression imputation
2023 have been proposed to enhance model performance in high-dimensional scttings by selectively encod- ing informative missing patterns. Many self-supervised learning techniques can be interpreted as training lo reconstruct the imputs with Knockout. In addition, Knockout can be trained with standard empirical risk minimization while some approaches need more complex optimization (Ma et al. 2021 2022). For example, masked language modcling (Devlin al. 2019) randomly maps tokens to an unseen "masked" token. De-
noising autoencoders (Vincent et al., 2010: randomly replace image patches with black patches, which are arguably out of the support of natural images.
4 Experiments
In all experiments, unless stated otherwise, we compare Knockout against common baseline model trained on complete data, which, at inference time, imputes missing variables with mean (if continuous) mode (if discrete) values. If the training is done incomplete data with observed missing variables, imputed with mean/mode, we denote this as common baseline*. For most results we report variant of Knockout but with sub-optimal placeholders (i.e. mean/mode for continuous /atuu////aeooiccal features). We de- note this variant as Knockout*. Note that both Knockout* and common bascline* use the same placeholder with the only difference being that Knockou models observe randomly knocked-out missingness in addition to (possible) observed missingness during training.
In all Knockout implementations,, mini-batches have no induced missing variable. Tf d is the number of input variables and is the knockout rate, we choose r such that the probability of not knocking out any variables in a mini-batch, iee. (1 (1-r)) is approximately 0.5. In batches with induced missingness, variables (or groups of variables in structured Knockout) are independently removed, with a probability equal to the knockout rate. The summary of the experimental sctups are shown in Table
choose random knockout rates such that, in expectation, half of the
4.1 Simulations
We perform regression simulations to predict the output from the input X R In each simulation, we sample 30k data points in total and use 10% for training. Since the input low-dimensional, we can additionally compare against sophisticated but computationally expensive imputation methods. The baselines for the simulation are:

---

Published in Transactions on Machine Learning Research (07/2025)
Impute Dropout
(Srivastava et al. 2014 Randomly sets subset of input features to zero during
training.
Zero Impute with Indicator: Replaces missing values with zeros and adds a binary indicator variable to denote the presence of missing data.
MICE (Van Buuren & Groothuis- Oudihioon 2011): Iteratively imputes missing values by modeling
each variable with missing data as a function of other variables in a round-robin fashion.
missForest (Stekhoven & Bühlmann 2012 :: Uses random forests to iteratively predict and fill in
missing values.
MIWAE Mattei & Frellsen 2019 : Imputes missing data. using a VAE trained with importance-
weighted bounds.
supMIWAE (Ipsen et al. 2022 : A supervised extension of MIWAE.
KNN Impute:
Fills in missing values by averaging the values of the k-nearest neighbors.
Linear Regression Impute: Imputes missing values using predictions from a linear regression model.
All methods use the same ncural network architecture composed of 3-layer multi-layer perceptron (MLP) with hidden layers 100 and ReLU activations. Training is done using Adam (Kingma & BBa 2014 with learning rate 3e-3 for 5k steps.
We generate training data corresponding to complete training data, MCAR training data, and MNAR training data. For MCAR data, each leature is independently missing with a probability of 10%. For MNAR data, we adopt the self-censored missing setup where a variable is missing if its value is above the variable 90th percentile. To ensure that performance comparisons reflect only differences in the training data, we keep the test data consistent across experiments. Specifically, during testing, pattern (i.e., a specific set of missing variables) uniformly to all samples in the test set. We repeat this procedure exhaustively for all missingness patterns involving up to 3 missing variables. This resulted in 130 different missing patterns.
apply predefined missingness
We evaluate the models' predictions against the MMSF-minimizing Bayes optimal predictions: X (please see Appendix B.1.1 for evaluation against observations Y). Fig. 2 shows the results of 10 repetitions of this simulation. Knockout outperforms all the baselines regardless of the types of training data (complete, MCAR, or MNAR). Dropout and Knockout* achieve similar performance but both are worse than Knockout; this underscores the importance of choosing an appropriate placcholder value. We also experimented with varying the training-test split size to simulate low-data and data-rich regimes in Appendix B.I.I
In addition, we perform an ablation to verify that choosing appropriate placeholders is critical for getting good performance (see Fig. 3). In this ablation, the value of the placeholder of Knockout is selected from the set (0, 2, 4, 6, 8, 10). After feature normalization, the mean value of a feature is 0 and using the placeholder value of 0 resulting in bad prediction performance (higher MSF). Choosing placeholder value that is away from 0 (i.e. away from the feature mean) lowers the test MSE in both complete data (Fig. 33) and missing data (Fig. 3p) setup. We also include an comparison aganist the common bascline with Z imputation in Appendix 1.1.2 to illustrate the benefit of Knockout training strategy. This baseline is similar to Knockout during inference but does not employ the augmentation strategy of Knockout during training.
To support our proposal for handling NMAR data, we include an ablation study comparing Knockout against its ablated version named Knockout-. While Knockout follows the procedure outlined in Section 22.4 Knockout- treats all observed data uniformly, without distinguishing NMAR obscrvations. Both methods were trained using the same MNAR data detailed previously. The results of this comparison are presented in Table 3 which reports the MSE across varying levels of missingness. Knockout consistently achieves lower MSEs, demonstrating the benefit of explicitly accounting for NMAR mechanisms.
4.2 Missing Clinical Variables in Alzheimer's Disease Forecasting
We demonstrate Knockout's effectiveness in handling observed missing data during a real-world clinical task: predicting whether patients with mild cognitive impairment. will progress to Alzheimer's Disease within the
8

---

Published in Transactions on Machine Learning Research (07 (07/2025)
a) Complete
b) MCAR
10
10
10
10 -2
10
Observations
Bayes Optimal
Observations
Bayes Optimal
T= 2 T
T 6
10
Figure 3: Lower test MSE in the regression simulation is achieved by selecting placeholder value that is far from the mean (i. 0). Observations: evaluate against Y. Bayes Optimal: evaluate against || [X].
Method
fm 0
fm 1
fm 2
fm
CB
0.361 + 0.183
2.953 + 3.870
4.053 +: 5.616
4.585 + 6.109
Knockout-
0.144 + 0.039
0.089 = 0.055
0.107 = 0.071
0.163 0.141
Knockout
0.095 + 0.044 0.072 + 0.058
0.095
0.062
0.175 0.156
Table 3: Knockout's vs. Knockout-'s 's MSE across varying numbers of missing features (fm) during inference.
next five years. We use data from the Alzheimer's Disease Neuroimaging Initiative (ADNT) database (Mueller ct al. 2005) and adopt the statc-of-the-art model from (Karaman al.) 2022). Input features X include subject demographics, genetics, and finical/imaging biomarkers. The target Y is binary vector and indicates AD diagnosis in each of the five follow-up years. For Knockout, we 10 for induced missingness and -10 for observed missingness, both during training and testing. Further details about the dataset and experimental setup are provided in Appendix B.2, Fig. shows the average AUROC scores when individual input. features are omitted during inference. These results are averaged over 10 random 80-20 train-test splits. For each split, we compute the AUROC by averaging AUROC scores across the five follow-up years. Knockout outperforms both the common bascline and Knockout in the vast majority of cases, cases, highlighting the importance of selecting an effective placeholder strategy.
out-of-range value of
4.3 Privileged Information for Noisy Label Learning
Knockout can be used to incorporate privileged information (PT) into the learning process. For instance, Pl might include demographic details that may be partially or completely missing during inference (Nguyen noisy label learning task, where the objective is to leverage additional Pl-such
et al. 2024b). We focus as the annotator's 's ID or the time of annotation to improve the model's 's robust against label noise. Due
to the absence of Pl in testing, existing methods (Ortiz-Jimenez et al. 2023 Wang et al. 2023 require an auxiliary classification head for PI utilization. Knockout can be directly applied with a method that accepts PI as input and achieve competitive performance. We follow previous experiment setups (Wang et al. 2023 and evaluated model performance on CIFAR-I0H (Peterson et al.] (2019) and CIFAR-10/TOON (Wei et al. 2021). These datasets involve relabeled versions of the original CIEAR. For more details, see Appendix B.5, As no-PI baseline, train Wide-ResNNet-10-2
2016 model that ignores Pl. We also compare against recent noisy label learning methods: НET (Collier al. 2021 and SOP Liu ct al. 2022). We implement Knockout with a similar architecture and training scheme as the no-PI bascline. where we concalenate the Pl with the image-derived features and randomly knock PI out during training, For the common baseline, we train the same architecture with complete training data, but mean imputation for PI data during inference. Table 3 lists test accuracy results. For the CIFAR-10H dataset, where we have high quality PI, Knockout outperforms all baselines by a large margin, improving test accuracy by 6%. For CIFAR-10/100 100N datasets, where have low quality PI during training, Knockout's 's boost is more modest.
(Zagoruyko & Komodakis
9

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Common baseline*
0.94
Knockout
0.93
Composite AUROC
0.92
0.89
PTEDUCAT
AGE
APOE4
CDR
FAG
ADAS11
ADAS13
ADAS04
RAVIT immediate
MMSE
RAVET leaming
RAVLT forgetting
RAVIT pere forgetting
LDELTOTAL
TRABSCOR
mPACCalgit
mPACCtrallss
ABETA
TAU
PTAU
FDG
Ventricles
Hippocanpus
WholeBrain
Entarhinal
Fusiform
MidTemp
ICV
Figure 4: AUROC scores obtained for the three model variants when each input feature is missing during inference (x-axis) in the Alzheimer's Disease forecasting experiment. Error bars indicate the standard error across 10 train-test splits.
Table 4: Average test accuracy (over 5 runs) of different methods on noisy label datasets with PI. For Quality, "High" and "Low" indicate access to sample-wise PI and batch-average PT respectively. Best results bold, socond-best underlined.
Datasets
Quality
No-P1
HET
SOP
Common baseline
Knockout
CIFAR-10H (Worst)
High
51.1+22
50.8
51.3-1.
55.2+o.s
57.4
CIFAR-10N (Worst)
Low
80.6+0.2
81.9+a.4
85.0+o.a
82.3+o.3
84.7+a.
CIFAR-100N (Fine)
Low
60.4
60.8
61.9-c.c
60.7
62.1+o.a
performing similarly with SOP and slightly better than HET and the no-PI baseline. Knockout can offer competitive results when we have access to high quality PI during training.
4.4 Missing Images in Tumor Segmentation
Here, we investigate the ability of Knockout to handle missingness in a high-dimensional, 3D dense image segmentation task. In particular, we experiment on a multi-modal tumor segmentation task (Baid et al., 2021), where the goal is delineate adult brain gliomas in 3D brain MRI volumes given 4 modalities per subject: T1, TIGd, T2, and FLAIR. We use 3D UNet the segmentation model (Ronneberger et al, 2015). We minimize sum of cross-entropy loss and Dice loss with equal weighting and use Adam optimizer with a learning rate of 1e-3. See Appendix B.3 and [.33 further details. At inference time, we evaluate on all modality missingness patterns. Fig. 5 shows Dice scores. We observe that the Knockout-trained model has better Dice performance across all missingness patterns. When all modalities are available, Knockout and the common baseline achieve the same performance level. See Appendix A.3 for additional experiments the effect of various placeholders.
4.5 Missing Views in Tree Genus Classification
We demonstrate Knockout's ability to deal with missing data at the latent feature level in a classification task. The Auto Arborist dataset (Beery et al. (022), multi-viww (strect and acrial) image dataset, used for this purpose. Tn this experiment, used the top 10 genera for multi class prediction. A frozen ResNet-
10

---

Published in Transactions on Machine Learning Research (07/202) 2025)
Common baseline
0.7
Dice
0.6
0.4
T2GAIR
AlI
T2+T2+FLAIR
TI+TIGd+FLAIR TI+TIGd+T2
T2+FLAIR
TIGd+FLAIR
T1Gd+12
T1+FLAIR
T1+T2
T1+TIGd
FLAIR
T2
TIGd
T1
Figure 5: Dice performance of molti-modal tumor segmentation across varying missingness patterns of modality images. Knockout trained models have better Dice performance across all missingness patterns than the common basclinc. Error bars depict the 95% confidence interval over test subjects.
Table 5: F1-scores of Auto Arborist averaged over 5 random seeds. Each column represents non-missing modalities at inference time. Best results in bold, second-best underlined.
Serial-Street
Aerial
Street
ResNet-50
Common baseline
0.483
0.31220.007
0.35640.024
Knockout, (Features)
0.493310.00
0.284 10.023
0.381 10.022
Knockout (Structured)
0.49640.016
0.308+0.024
0.41640.014
ViT-B-16
Common baseline
0.46440.018
0.305=0.022
0.388
Knockout (Features) Knockout (Structured)
0.473
0.315+0.00s
0.383
0.480.10.017
0.324
0.408 10.015
50 (He et al. 2016) and ViT-B-16 (Dosovitskiy et al. 2021 pretrained with ImageNet-v2 (Recht et al. 2019 is used feature extractor. The two features from street and acrial images are concatenated and were fed into 3-layer MLP with ReLU activations. We trained Knockout to randomly replace the whole latent vectors with vectors of Os as placeholders after normalization. This variant is denoted as Knockout (Structured). We additionally trained two baselines for comparison: 1) Knockout (Features) where individual features in the latent vectors are independently replaced with placeholders, and 2) an imputation baseline, substituting latent vectors from missing views with vectors of zeros during inference. Table Knockout (Structured) putperfrringg Knockout (Features), suggesting that matching p(M) with missing patterns that we expect to see at inference can be more effective.
4.6 Missing MR Modalities in Prostate Cancer Detection
We demonstrate structured Knockout in the context of a binary image classification task, where Knockout is applied at the latent level. The dataset consists of T2-weighted (T2w), diffusion--wighhted (DWI) and apparent diffusion coefficient (ADC) MR images per subject (Saha et al.
2022 A simple "ensemble baseline"
approach to address missingness is train separate convolutional classifier for each modality, and average the predictions of available modalities at inference time (Kim al., 2023 Hu al. 2020).. For further details, please see Appendix B.4 To train model with latent-level structured Knockout, we use the same 3 feature extractors. Each feature extractor is trained with different modality. The loss function is binary cross entropy loss and we use an Adam optimizer with lcarning rate of 1c-3. We randomly knock out each modality. For the common baseline, we trained the same archilecture with complete modalities. At inference time, the latent features from missing modalities are imputed with Os. In the "ensemble baseline" approach, we averagod the prodicted values from the three extractors without additional training. As shown in Table Knockout generally outperforms the baselines in the majority of scenarios, except for inputs with ADC. Notably, the F1 scores from the popular ensemble baseline are significantly lower than Knockout.
11

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 6: F1 scores of prostate cancer dataset averaged over 5 random seeds across varying missingness patterns at inference time. Each column represents non-missing modalities. Best results in bold, second- best underlined.
ADC
T2
T2
T2
ADC
DWT
All
+DWI
+ +DWI
+ADC
Ensemble 2.21
0.37
0.28
0.32
0.18
0.33
0.30
Common 2.43 0.68 10.02
0.61
10.0z
0.70
0.51 20
0.65
0.67 20.01
Knockout 0.63 0.60+
0. 0.62 +0.02
0.67+a.am
0.66+a.0l
0.65+o.02
0.68+0.01
4.7 Missing Modalities in UPMC Food-101 Classification
To further evaluate the generalizability of our approach, extend experiments to the UPMC Food-101 dataset (Wang et al., 2015), widely used benchmark for multi-modal learning with both textual and visual inputs. Our multi-modal architecture combines pretrained ResNet50 (vision) and RoBERTa-base (text) backbones. The output from ResNet50 is fed into a single linear layer to transform its dimension to match that of Roberta's output. Feature vectors from each modality are concatenated and passed through an MLP with two hidden layers of size 256 and ReLU activations. Models are trained using Adamm with a learning of 3 10 10-5 for epochs and batch size of 72. We follow the standard train-test split, reserving 7,000 samples from the training set for validation.
As shown in Table 7 our multi-modal model with Knockout achieves 92.7% accuracy, outperforming the common Baseline (91.5%) and matching the previous state-of-the reported by (Gallo et al. 2020 Table 7 also reports performance under unimodal ablations. Knockout exhibits greater robustness, especially in the image-only setting, improving accuracy from 47.5% to 63.2%.
Table 7: Test accuracy (%) on the Food-101 dataset for multi-modi models under three input settings: (1) both Image and Text, (2) Image only (without Text), and (3) Text only (without Image). For comparison, test accuracies of multi-modal models from previous work are also included.
Method
Image Model
Text Model
Image + Text
w/o Text
w/o Image
Wang al. 2015
VGG19
TF-IDF
85.1
N/A
N/A
Kiela et al.
2018
ResNet152
Fext
90.8
N/A
N/A
Gallo et al.
2020
Inception-v3
BERT-LSTM
92.5
N/A
N/A
Common baseline
ResNet550
RoBERTa
91.5
47.5
85.0
Knockout
ResNet50
RoBERTa
92.7
63.2
85.3
5 Discussion
We introduced Knockout, novel, easy-to-implement strategy for handling missing inputs, using math- ematically principled approach. By simulating missingness during training via random "knock out" and substitution with appropriate placeholder values, our method allows single model learn the conditional distribution and all desired marginals. Our evaluation demonstrates Knockout's 's the versatility and robust- ness across diverse synthetic and real-world scenarios, consistently outperforming conventional imputation and ensemble techniques for both low and dineesiinnio missing inputs. We also extend Knockout to handle observed missing values in training and present a structured version that is more effective when entire feature vectors or input modalities are missing.
There has been a growing focus on developing multimodal datasets, tasks, and models that integrate in- formation across modalities improve performance and generalization Ngiaa al. (2011): Srivastava & Salakhutdinov (2012): Nguyen et al. (2018); Lu al.) (2019); Alayrac et al. (2022): Boocking et al. (2022). motivating continued efforts to improve robustness to missing or incomplete inputs. Although modern mul-
12

---

Published in Transactions on Machine Learning Research (07/2025)
timodal models are increasingly tolerant of missing modalities, Knockout can still play crucial role in promoting robustness and adaptability. This is especially useful during deployment, where models often face unpredictable or incomplete imputs due to sensor failures or missing data. Multimodal foundation models like LLaVA (Liu et al.) 2023) and VILA {Lin et al., 2024 are generally more robust to missing modalities than earlier models, but they are not immune to failure. Their ability to handle missing inputs largely stems from pretraining on massive, diverse datasets. However, when applied to tasks with input distributions that differ significantly from their training data, these models may still struggle in the absence of a modality, Even SOTA transformer-based models have been shown to be sensitive to missing modalities (Ma et al. 2022
There are several future directions for further investigation. While our paper highlights the importance of choosing an appropriate placeholder value, and there appears to be a practical tension between selecting an unlikely/infeasible value versus achieving numerical stability (e.g., avoiding exploding gradients), one can conduct more detailed study of this to optimize the placcholder value. Additionally, our method assumes model training with SGD, where samples are seen multiple times. Extending it to models that do not use SGD, such as XGBoost, is valuable avenue. Another promising direction of future research is adapting Knockout to address distribution shifts in the presence of missingness. Finally, Knockout's theoretical treatment hinges on the use of a high capacity, non-lime linear model trained on very large data. In applications, where low capacity models are used and/or training data are limited, Knockout might not be as effective.
Acknowledgments
We thank Rachit Saluja, Leo Milecki, Haomiao Chen, and Benjamin C. Lee for the helpful comments. Funding for this project was in part provided by the NIH grants RO1AG053949, ROLAG064027, ROLAG07098S and 1K25CA283145, and the NSF CAREER 1748377 grant.
Data collection and sharing for this project was funded by the Alzheimer's Disease Neuroimaging Initiative (ADNI) (National Institutes of Health Grant U01 AG024904) and DOD ADNI (Department of Defense award number W11XWWH-12---0012). ADNI is funded by the National Institute on Aging, the National Institute of Biomedical Imaging and Bioengineering, and through generous contributions from the following: AbbVie, Alzheimer's Association; Alzheimer's 's Drug Discovery Foundation; Araclon Biotech; BioClinica, Inc.; Biogen; Bristol-Myers Squibb Company; CereSpir, Inc.; Cogstate; Eisai Inc.; Elan Pharmaceuticalss, Inc.; Eli Lilly and Company; Eurolmmm;; F. Hoffmann- La Roche Ltd and its affiliated company Genentech, Inc.; Fujirebio;
GE Healthcare; IXICO Ltd.; Janssen Alzheimer Immunotherapy Research & Development, LLC.; Johnson & Johnson Pharmaceutical Research & Development LLC.; Lumosity; Lundbeck; Merck & Co., Inc.; Meso Scale Diagnosties, T.I.C.; NeuroRx Research; Neurotrack Technologies; Novartis Pharmaceuticals Corporation; Pfizer Inc.; Piramal Imaging; Servier; Takeda Pharmaceutical Company; and Transition Therapeutics. The Canadian Institutes of Health Research is providing funds to support ADNI clinical sites in Canada. Private sector contributions are facilitated by the Foundation for the National Institutes of Health (www.fnih. www.fnih.orgg) The grantee organization is the Northern California Institute for Research and Education, and the study is coordinated by the Alzheimer's Therapeutic Research Institute at the University of Southern California. ADNI data are disseminated by the Laboratory for Neuro Imaging at the University of Southern California.
References
Jean-Baptiste Alayrac, Jeff Donahne, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Proceedings of NeurIPS, 35:23716-2351 237366 2022.
Ujjwal Baid, Satyam Ghodasara, Suyash Mohan, Michel Bilello, Evan Calabrese, Errol Colak, Keyvan Fara- hani, Jayashree Kalpathy-Crames Felipe C. Kilamura, Sarthak Pati, Luciano M. Prevedello, Jeffrey D. Rudie, Chiharu Sako, Russell T. Shinohara, Timothy Bergquist, Rong Chai, James Eddy, Julia Elliott, Walter Reade, Thomas Schaffter, Thomas Yu, Jiaxin Zheng, Ahmed W. Moawad, Luiz Otavio Coelho, Olivia McDonnell, Elka Miller, Fanny E. Moron, Mark C. Oswood, Robert Y. Shih, Loizos Siakallis, Yulia Bronstein, James R. Mason, Anthony F. Miller, Gagandeep Choudhary, Aanchal Agarwal, Cristina H. Be-
13

---

Published in Transactions on Machine Learning Research (07 (07/2025)
sada, Jamal J. Derakhshan, Mariana C. Diogo, Daniel D. Do-Dai, Luciano Farage, John L. Go, Mohiuddin Hadi, Virginia B. Hill, Michael Iv, David Joyner, Christie Lincoln, Eyal Lotan, Asako Miyakoshi, Mariana Sanchez-Montano, Jaya Nath, Xuan V. Nguyen, Manal Nicolas-Jilwan, Johanna Ortiz Jimenez, Kerem Oz- turk, Bojan D. Petrovic, Chintan Shah, Lubdha M. Shah, Manas Sharma, Onur Simsck, Achint K. Singh, Salil Soman, Volodymyr Slatsevych, Brent D. Weinberg, Robert J. Young, Ichíro Ikuta, Amit K. Agarwal, Sword C. Cambron, Richard Silbergleit, Alexandru Dusoi, Alida A. Postma, Laurent Letournean-GGuilonn Gloria J. Guzman Perez-Carrillo Atin Saha, Neetu Soni, Greg Zaharchuk, Vahe M. Zohrabian, Yingming Chen, Milos M. Cekic, Akm Rahman, Juan E. Small, Varun Sethi, Christos Davatzikos, John Mongan, Christopher Hess, Soonmee Cha, Javier Villanueva-Meyer, John B. Freymann, Justin S. Kirby, Benedikt Wiestler, Priscila Crivellaro, Rivka R. Colen, Aikaterini Kotrotsou, Daniel Marcus, Mikhail Milchenko, Arash Nazeri, Hassan Fathallah--Shakk,, Roland Wiest, Andras Jakab, Marc-Andre Weber, Abhishek Ma- hajan, Bjoern Menze, Adam E. Flanders, and Spyridon Bakas, The rsna-asmi ssnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification, 2021.
Sara Beery, Guanhang Wu, Trevor Edwards, Filip Pavetic, Bo Majewski, Shreyasce Mukhorjee, Stanley Chan, John Morgan, Vivek Rathod, and Jonathan Huang. The auto arborist dataset: a large-scale benchmark for multiview urban forest monitoring under domain shift. In Proceedings of CVPR, pp. 11194-21300 2022.
Mohamed Belghazi, Maxime Oquab, and David Lopez-Paz. Learning about an exponential amount of conditional distributions. Proceedings of NewrIPS, 32, 2019.
Dimitris Bertsimas, Arthur Delarue, and Jean Pauphilet. Simple imputation rules for prediction with missing data: Theoretical guarantees vs. empirical performance. TMLR, 2024.
Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez-Valla et al. Making the most of text semantics to improve biomedical vision-language processing. In Proceedings of ECCV, pp. 1-21.
Springer, 2022.
Philippe Brouillard, Sébastien Lachapelle, Alexandre Lacoste, Simon Lacosto-Julien, and Alexandre Drouin. Dillerentiable causal discovery from interventional data. In Proceedings of NewrIPS, volune 33, pp. 21865 21877, 2020.
Rich Caruana. Multitask learning. Machine learning, 28:41-78 75, 1997.
Minmin Chen, Kilian Weinberger, Fei Sha, and Yoshua Bengio. Marginalized denoising auto-encoders for nonlinear representations. In Proceedings of ICML, pp. 1476 1484. PMLR, 2014.
Mark Collier, Basil Mustafa, Efi Kokiopoulou, Rodolphe Jenatton, and Jesse Berent. Correlated input- dependent label noise in large-scale image classification. In Proceedings of CVPR, pp. 1551 1560, 2021.
Rahul S. Desikan, Florent Ségonne, Bruce Fischl, Brian T. Quinn, Bradford C. Dickerson, Deborah Blacker, Randy L. Buckner, Anders M. Dale, R. Paul Maguire, Bradley T. Hyman, Marilyn S. Albert, and Ronald J. Killiany. An automated labeling system for subdividing the human cerebral cortex on mri scans into gyral based regions of interest. NeuroImage, 31(3):968 980, 2006. ISSN 1053-8119. doi: DOI:10.1016/j. neuroimage.2006
URL http:
sci httt:://www... //www
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL, pp. 4171 4186, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa. Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. In Proceedings of ICLR, 2021.
Bruce Fischl, André van der Kouwe, Christophe Destricux, Eric Halgren, Florent Ségonne, David H. Salat, Fvellna. Busa, Larry J. Seidman, Jill Goldstein, David Kennedy, Verne Caviness, Nikos Makris, Bruce
14

---

Published in Transactions on Machine Learning Research (07/2025)
Rosen, and Anders M. Dale. Automatically Parcellating the Human Cerebral Cortex. Cerebral Corter, 14 (1):11-22, 2004. doi: (00003//eeror//ggggge URT. 1/11.abstract
Yarin Gal and Zoubin Ghahramani. Dropout as Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In Proceedings of ICML, pp. 1050-1059. PMLR, 2016.
Ignazio Gallo, Gianmarco Ria, Nicola Landro, and Riccardo La Grassa. Image and text fusion for upme food-101 using bert and cnns. In International Conference on Image and Vision Computing New Zealand. TEEE, 2020.
Lovedeep Gondara and Ke Wang. Mida: Multiple imputation using denoising autoencoders. In Proceedings of PAKDD, pp. 260 272. Springer, 2018.
Elad Hazan, Roi Livni, and Yishay Mansour. Classification with low rank and missing data. In Proceedings of ICML, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of CVPR, pp. 770 770-778, 2016.
Dan Hu, Han Zhang, Zhengwang Wu, Fan Wang, Li Wang, J Keith Smith, Weili Lin, Gang Li, and Ding- gang Shen. seentangled-multioodd adversarial autocncoder: Application to infant age prediction with incomplete multimodal neuroimages. IEEE TMI, 39(12)44137 4149,, 2020.
Niels Bruun Ipsen, Pierre-Alexandre Mattei, and Jes Frellsen. How to deal with missing data in supervised deep learning? In Proceedings of TCI.R, 2022.
Oleg Ivanov, Michael Figurnov, and Dmitry Vetrov. Variational autoencoder with arbitrary conditioning. In Proceedings of ICLR, 2019.
Hailey James, Chirag Nagpal, Kathorine A Heller, and Berk Ustun. Participatory personalization in classi- fication. In Proceedings of NeurIPS, 2023.
Julie Josse, Nicolas Prost, Erwan Scornet, and Gaël Varoquaux. On the consistency of supervised learning with missing values. arXix: 06931, 2019.
Batuhan K. Karaman, Elizabeth C. Mormino, and Mert R. Sabuncu. Machine learning based multi-modal prediction of future decline toward alzheimer's discase: An empirical study. PLOS ONE, 17:c0277322, 11 2022. doi: 10.1371/journal.poor 02773327
Nan Rosemary Ke, Olexa Bilaniuk, Anirudh Goyal, Stefan Bauer, Hugo Larochelle, Bernhard Schölkopf, Michael Curtis Mozer, Christopher Pal, and Yoshua Bengio. Neural causal structure discovery from interventions. TMLR, 2023.
Douwe Kiela, Edouard Grave, Armand Joulin, and Tomas Mikolov. Efficient large-scale multi-modal classi- fication. In Proceedings of AAAI, volume 32, 2018.
Heejong Kim, Daniel JA Margolis, Himanshu Nagar, and Mert R Sabuncu. Pulse sequence dependence of a simple and interpretable deep learning method for detection of clinically significant prostale cancer using multiparametric mri. Academic Radiology, 30(5):966-9)) 99009666677,,0 2023.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Proceedings of ICLR, 2014.
Trent Kyono, Yao Zhang, Alexis Bellot, and Mihaela van der Schaar. Miracle: Causally-aware imputation via learning missing data mechanisms. In Proceedings of NewrIPS, 2021.
Ranjit Lall and Thomas Robinson. The midas touch: accurate and scalable missing-data imputation with deep learning. Political Analysis, 30(2):179-1966 2022.
15

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Marine Le Morvan, Julie Josse, Thomas Moreau, Erwan Scornet, and Gaël Varoquaux. Neumiss networks: differentiable programming for supervised learning with missing values. In Proceedings of NewrIPS, 2020a.
Marine Le Morvan, Nicolas Prost, Julie Josse, Erwan Scornet, and Gael Varoquaux. Linear predictor on linearly-generated data with missing values: non consistency and solutions. In Proceedings of AISTATS. PMLR, 2020b.
Marine Le Morvan, Julie Josse, Erwan Scornet, and Gaöl Varoquaux. What's a good imputation to predict with missing values? In Proceedings of NewrIPS, 2021.
Steven Cheng-Xian Li, Bo Jiang, and Benjamin Marlin. Misgan: Learning from incomplete data with generative adversarial networks. In Proceedings of ICLR, 2019.
Yang Li, Shoaib Akbar, and Junier Oliva. Acflow: Flow models for arbitrary conditional likelihoods. In Proceedings of ICML, pp. 5831 5841. PMLR, 2020.
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. VILA: On Pre-training for Visual Language Models. In Proceedings of CVPR, pp. 18689-26699 2024.
Phillip Lippe, Taco Cohen, and Efstratios Gavves. Efficient neural causal discovery without acyclicity constraints. In Proceedings of ICLR, 2022.
Roderick JA Little and Donald B Rubin. Statistical analysis with missing data, volume 793. John Wiley & Sons, 2019.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Proceedings of NewrIPS, 96:34892-34916, 2023.
Sheng Liu, Zhihui Zhu, Qing Qu, and Chong You. Robust training under label noise by over over- pee-pprramtterizatoon In Proceedings of TCML, pp. 44153-14172 PMLR, 2022.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for dision-and-langune tasks. Proceedings of NewrIPS, 32, 2019.
Chao Ma, Sebastian Tschiatschek, Konstantina Palla, Jose Miguel Hernandez- Lobato, Sebastian Nowozin, and Cheng Zhang. Eddi: Efficient dynamic discovery of high-value information with partial vae. In Proceedings of ICML, 2019.
Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng. Smil: Multimodal learning with severely missing modality. In Proceedings of AAAT, 2021.
Mengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng. Are multimodal transformers robust to missing modality? In Proceedings of CVPR, 2022.
Laurens Maaten, Minmin Chen, Stephen Tyree, and Kilian Weinberger. Learning with marginalized cor- rupted features. In Proceedings of ICML, pp. 410 418. PMLR, 2013.
Pierre-Alexanne Mattei and Jes Frellsen. Miwae: Deep generative modelling and imputation of incomplete data sets. In Proceedings of ICML, pp. 4413 4123.423. PMLR, 2019.
Susanne G. Mueller, Michael W. Weiner, Leon J. Thal, Ronald C. Petersen, Clifford R. Jack, William Jagust, John Q. Trojanowski, Arthur W. Toga, and Laurel Beckett. Ways toward an early diagnosis in alzheimer's disease: The alzheimer's disease neuroimaging initiative (adni). Alzheimer's & Dementia, 1:55-66 66, 07 2005. doi: jal.6//.al..2005..666
Natalia Neverova, Christian Wolf, Graham Taylor, and Florian Nebout. Moddrop: adaptive multi-modal gesture recognition. IEEE TPAMI, 3898):1692-1706, 2015.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, Andrew Y Ng, et al. Multimodal deep learning. Tn Proceedings of ICML, volume 11, pp. 689 696, 2011.
16

---

Published in Transactions on Machine Learning Research (07/2025)
Minh Nguyen and Mert R. Sabuncu. Efficient Identification of Direct Causal Parents via Invariance and Minimum Error Testing. TMLR, 2024.
Minh Nguyen, Gia Ngo, and Nancy Chen. Multimodal neural pronunciation modeling for spoken languages with logographic origin. In Proceedings of EMNLP, pp. 2916 2016,2922, 2018.
Minh Nguyen, Tong He, Lijun An, Daniel C Alexander, Jiashi Feng, BT Thomas Yoo, Alzheimer's Dis- ease Neuroimaging Initiative, et al. Predicting alzheimer's disease progression using deep recurrent neural networks. Neurolmage, 222: 117203, 2020.
Minh Nguyen, Gia Ngo, and Mert R. Sabuncu. GLACIAL: Granger and starning-based Causality Analysis for Longitudinal Studies. The Journal of Machine Learning for Biomedical Imaging, 2024a.
Minh Nguyen, Alan Q. Wang, Heejong Kim, and Mert R. Sabuncu. Adapting to Shifting Correlations with Unlabeled Data Calibration. In Proceedings of ECCV, 2024b.
Guillermo Ortiz-Jimenez, Mark Collier, Anant Nawalgaria, Alexander Nicholas D'Amour, Jesse Borent, Rodolphe Jenatton, and Ef Kokiopoulou. When does privileged information explain away label noise? In Proceedings of ICML, pp. 26646 26669, PMLR, 2023.
Srinivas Parthasarathy and Shiva Sundaram. Training strategies to handle missing modalities for audio-visual expression recognition. In International Conference on Multimodal Interaction, 2020.
Ignacio Peis, Chao Ma, and José Miguel Hernández-Lobato Missing data imputation and acquisition with
deep hicrarchical models and hamiltonian monte carlo. In Proceedings of NeurIPS, 2022.
Joshua C Peterson, Ruairidh M Battleday, Thomas L Griffiths, and Olga Russakovsky. Human uncertainty makes classification more robust. In Proceedings of ICCV, pp. 9617-9626, 2019.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize imagenet? In Proceedings of ICML, pp. 5389-5400, 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of MICCAI, pp. 234 241. Springer, 2015.
Anindo Saha, Jasper Jonathan Twilt, Jocran Sander Bosma, Bram van Ginneken, Derya Yakar, Mattijs Elschot, Jeroen Veltman, Jurgen Fütterer, Maarten de Rooij, and Henkjan Huisman. The pi-cai challenge: public training and development dataset, 2022.
P Spirtes, C Glymour, R Scheines, and David Heckerman. Causation, prediction, and search. MIT press, 2000.
Nitish Srivastava and Russ Salakhutdinov. Multimodal learning with deep boltzmann machines. Proceed- ings of NewrIPS, 25, 2012.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. JMLR, 15(1):1929 1958, 2014.
Daniel J Stekhoven and Peter Bühlmann. type data. Bioinformatics, 28(1):112 118, 2012.
non-parmett--....riri.... missing value imputation for mixed-
Olga Troyanskaya, Michael Cantor, Gavin Sherlock, Pat Brown, Trevor Hastic, Robert Tibshirani, David Botstein, and Russ B Altman. Missing value estimation methods for dna microarrays. Bioinformatics, 17 (6):520-522) 525, 2001.
Baris Turkbey, Andrew B Rosenkrantz, Masoom A Haider, Anwar Pachani, Geert Villeirs, Katarzyna J Macura, Clare M Tempany, Peter L Choyke, Francois Cornud, Daniel J Margolis, et al. Prostate imaging reporting and data system version 2.1: 2019 update of prostate imaging reporting and data system version 2. Eauropean urology, 76(3):340-355 2019.
17

---

Published in Transactions on Machine Learning Research (07/2025)
Stef Van Buuren and Karin Groothuis- mice: Multivariate imputation by chained equations in
r. Journal of statistical software, 45:1-61 2011.
Mike Van Ness, Tomas M Bosschieter, Roberto SaappinGregorioo and Madeleine Udell. The Missing Indicator Method: From Low to High Dimensions. In Proceedings of KDD, pp- 5004-5015, 2023.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, vol- ume 47. Cambridge university press, 2018.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with local denoising criterion. JMLR, 11(12), 2010.
Ke Wang, Guillermo Ortiz-Jimenez, Rodolphe Jenatton, Mark Collier, Efi Kokiopoulon, and Pascal Frossard. Pi-dual: Using privileged information to distinguish clean from noisy labels. orXiv preprinl wrXiv:2310006000 2023.
Xin Wang, Devinder Kumar, Nicolas Thome, Matthieu Cord, and Frederic Precioso. Recipe recognition with large multimodal food dataset. In IEEE ICMEW, 2015.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: study using real-world human annotations. arXie preprint arXiv:21 12088, 2021.
Jinsung Yoon, James Jordon, and Mihacla Schaar. Gain: Missing data imputation using generative adver- sarial nets. In Proceedings of ICML, pp. 5689 5698. PMLR, 2018.
Jiaxuan You, Xiaobai Ma, Yi Ding, Mykel Kochenderfer, and Jure Leskovec. Handling missing data with graph representation learning. In Proceedings of NewrIPS, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of BMVC, 2016.
18


Reviews:
[{"review_id": "v2QXqM4vw3", "reviewer": "Reviewer_K3oz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Addresses a common and important problem in machine learning (missing values)\n- Proposes a simple, easy-to-implement approach to handling missing inputs during inference that requires minimal changes to existing training frameworks. Specifically, it avoids the computational overhead of training separate models or implementing complex imputation strategies.\n- Presents a theoretically grounded treatment with clear mathematical justification for its ability to learn both conditional and marginal distributions simultaneously.\n- Provides some practical guidance on selecting appropriate placeholder values for different data types and dimensionalities. \n- Includes experiments that address both missing completely at random (MCAR) and missing not at random (MNAR) scenarios\n\nWeaknesses:\n- Limited baseline comparison for missing data imputation methods. While sophisticated methods like MIWAE, supMIWAE, MICE, and missForest are included, simpler yet effective methods like KNN imputation are oddly absent. KNN imputation is computationally efficient and often performs well on continuous data. Similarly, basic regression imputation methods should have been included as they form important baselines for both the simulated and real-world experiments.\n- Strangely small training dataset size for simulated experiments. Using only 3,000 points (10% of 30k) for training is surprisingly small for a simple regression task with only 9 input dimensions, especially since several of the comparison methods rely on large models. This raises concerns that performance differences may be artificially magnified due to data scarcity, and also strangely contradicts the paper's later claim that \"Knockout's theoretical treatment hinges on the use of a high capacity, non-linear model trained on very large data.\"\n- Insufficient training details for comparison methods. The paper doesn’t include any implementation details for baselines (e.g., number of trees/max depth for random forest methods, training parameters for MIWAE/supMIWAE, iterations for MICE).\n- Theoretical and empirical inconsistency regarding placeholder values. The paper presents theory suggesting out-of-support placeholder values are optimal, but experiments suggest rare but in-support values work better. This contradiction is not adequately resolved, which is problematic since placeholder value selection is presented as one of the paper's main contributions.\n- Unexplained discrepancy in Figure 2 performance. With complete training data and 0 missing variables at test time (where no imputation occurs), different methods still show significantly varying performance. Why is this? Shouldn’t all methods perform the same in this scenario since no imputation is happening?\n- Incomplete reporting of the covariance matrix generation process. Although Appendix B.1.1 states Σ is drawn as Σ = WᵀW with W ~ Uniform(0, 1)¹⁰ˣ¹⁰, the paper fails to report whether Σ is fixed or changes across iterations. This is crucial because Σ affects signal-to-noise ratios and feature correlations, potentially disadvantaging predictive imputation methods. At minimum, the fixed value or the mean and eigen-spectrum should be reported.\n- Incomplete information on training data missingness rates. The fraction of missing data in the training sets for MCAR and MNAR scenarios in the simulated data experiments is never specified. While test scenarios vary from 0-3 missing variables (out of 9), the training missing data rate remains unquantified. This is important because methods requiring complete cases (common-baseline, dropout) will have reduced effective sample sizes by an unknown amount.\n- Omission of standard benchmark datasets. The paper excludes widely used datasets like MIMIC in favor of more specialized datasets, limiting comparison with existing literature.\n- Unclear justification for knockout implementation choices. The paper states \"In all Knockout implementations, we choose random knockout rates such that, in expectation, half of the mini-batches have no induced missing variable\" without explaining the reasoning behind this design choice.\n- Absence of subgroup impact analysis. Many works in this area investigate how missingness and imputation affect model performance across demographic or clinical subgroups. The paper lacks any discussion on experiments addressing this concern.\n- Limited discussion of related work. The paper does not adequately discuss its relationship to highly related work such as \"The Missing Indicator Method: From Low to High Dimensions.\" Similarly the paper states “Denoising autoencoders (Vincent et al., 2010) randomly replace image patches with black patches, which are arguably out of the support of natural images.” Isn’t using out of support values the exact theoretical result proposed?\n\nMinor/Nit: \n- Insufficient description of baseline methods. Section 4 merely lists acronyms without sketching how MIWAE, supMIWAE, missForest, MICE, Dropout, ZI actually work or why they're appropriate benchmarks. Each method should have at minimum a one-sentence principle, a reference, and justification for its hyper-parameter choices.\n- Typos (e.g., \"Knockoutto\")", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- A simple and efficient approach called \"Knockout\" for handling missing input features during inference without requiring multiple models or complex imputation techniques\n- A theoretical framework demonstrating that Knockout implicitly learns both conditional distributions (with full inputs) and marginal distributions (with missing inputs) simultaneously\n- Practical guidance for selecting appropriate placeholder values based on data type and dimensionality (categorical, continuous, structured inputs like images)\n- Empirical demonstration of Knockout's effectiveness across diverse scenarios", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical for recommendation to accept:\n- Add comparison to simple baselines (e.g., knn imputation, regression imputation) for all experiments.\n- Perform clear and fair comparisons with alternative methods. For simulations, this should include experiments where the baseline performance (without missingness) is the same or very similar. The training setup should not unfairly disadvantage predictive imputation methods (e.g., methods that require large training sets or correlations between features). To be clear, outperforming all baselines in every scenario would not be required for my recommendation.\n- Report all training information across all baseline methods. E.g., missingness rate, training procedure for comparison methods, covariance matrix for simulation experiments, etc).\n- Expand the theoretical and experimental investigation to justify and evaluate the impact of different placeholder value choices.\n\n\nWould strengthen the work:\n- Release the code required to reproduce the experimental results\n- Rework the related work section to more clearly discuss relationships to relevant papers, add missing citations.\n- Explain and justify design choices (e.g., half of all the mini batches).\n- Add standard benchmark datasets (e.g., MIMIC)\n- Experiments on the impact of subgroups given missingness and knockout imputation.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZyVomL4zao", "reviewer": "Reviewer_6JtJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The method is simple to understand and to implement\n- It is versatile, in that it applies to a large range of predictive models and tasks\n- It is effective, achieving competitive performance on a wide range of numerical examples\n\nWeaknesses \n- Inconsistent theoretical underpinning. The theoretical justification provided in the paper (and Appendix A) is light and not always rigorous. At the same time, existing results from the literature could be referenced/used more precisely to better support the proposal\n- Treatment of observed missingness during training (Section 2.4). This is a very important case as, in practice, most training data contains missing entries as well. I find the current treatment of NMAR not very realistic, because they assume that observations for which data is missing NMAR are flagged a priori, both in training and testing. \n- The numerical experiments do not provide the sufficient level of details to (a) appreciate the relative contribution of each component of the approach; (b) fully understand the experimental setup and the results presented.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a method, called “knockout” to handle missing inputs, especially for multi-modal or high-dimensional inputs. \nTheir method relies on two key components:\n(C1) imputing missing values with an out-of-range (OOR) value $\\bar{x}$\n(C2) at training, randomly ampute+impute some of observations used in training. Amputation is performed according to a MCAR model.\nExisting theoretical evidence suggests that simple imputation rule like OOR can be asymptotically consistent. However, to achieve this performance, it is central that (i) the predictor is trained on imputed data so that it can be learn how missing values are encoded; (ii) the missingness pattern is consistent in training. \nOn this perspective, (C1) is consistent with existing theory, while (C2) guarantees that (i) holds.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Scope: Currently, the method described applies to methods that are trained via stochastic gradient descent so that the amputation is theoretically harmless (in a SGD implementation where each sample can be used infinitely many times, it is okay to randomly ampute it, as long as the fraction of time it is left untouched is >0). (i) It would be good for the authors to state this limitation/requirement more clearly; (ii) The authors could consider an extension of their approach to methods that violate these assumptions (e.g., XGBoost) by either replacing some observations by an amputed+imputed version or duplicating some observations and amputing+imputing their duplicate. \n\n- Theoretical justification for out-of-range prediction: The authors provide some theoretical reasoning  for using out-of-range prediction in Appendix A.1 and A.2. This reasoning is very informal and partially flawed (in Appendix A.2, it is not clear if p is a density or a probability, in which case Eq. (16) could have 0 in the denominator). However, some of the cited works\n+ Julie Josse, Nicolas Prost, Erwan Scornet, and Ga.l Varoquaux. On the consistency of supervised learning with missing values. arXiv:1902.06931, 2019.\n+ Dimitris Bertsimas, Arthur Delarue, and Jean Pauphilet. Simple imputation rules for prediction with missing data: Theoretical guarantees vs. empirical performance. TMLR, 2024.\nshow that both mean imputation (for continuous variables) and out-of-range imputation (for both continuous and discrete variables) can be optimal in a setting with infinite data. As far as I can tell, the only reason to prefer OOR would be finite-sample empirical performance (which is a very solid justification to me, but should be stated as such). I think proper reference to these theoretical results (that support of both OOR and mean) and an honest justification of the reasons for the authors to prefer OOR in their work would be convincing that the half-baked theoretical arguments presented currently.\n\n- Fair common baseline: In light of the theoretical results stated above, I believe the common baseline should do impute missing categorical variables as its own category (or all 0s if one-hot encoded) as mode imputation is not asymptotically consistent. Same remark for Knockout*.\n\n- p.6 Section 4 “we choose random knockout rates such that, in expectation, half of the mini-batches have no induced missing variable.” It would be good to provide some numbers (orders of magnitude in the main paper, actual values in Appendix) of what these knockout rates are. It would also help the reader implement knockout for their problems (even though the math is fairly elementary, readers might appreciate a formula to pick p as a function of dimension, mini batch size, etc..)\n\n- Details on experiments in Section 4.1: I did not understand how the authors handled the missingness pattern: \n+ They mention “training data corresponding to complete training data, MCAR training data, and MNAR training data”. Does it mean the test data follows the same missingness pattern? Or is complete case only? \n+ In the caption of Fig 2: “X axis indicates the number of missing variables at inference time”  What does this mean? Are they stratifying the test set based on |{ j : M_j = 1}| or are they only allowing a subset of features (0,1,2,3) to be missing in testing? In training as well? \n+ I assume that, for MCAR, each feature is missing with probability 10%, isn't it? \n+ For MNAR, did the authors implement their two-vector approach from Section 2.4? \nThese details should be more clearly presented. \n\n- Results in Section 4.1 depending on training/testing patterns: It would be insightful to report the results of Section 4.1 for different training sets and testing sets: Namely for each training set (complete, MCAR, MNAR), report the MSE on a test set with complete data or MCAR or MNAR. I am thinking of 3 grouped bar plot. One plot = one training data set. One group = one testing set. And one bar per method. I think this will help support the intuition that the MCAR amputation at training induced by the knockout is not too harmful out-of-sample, even when the distribution in the testing set is not MCAR (i.e., either no missing data in testing or MNAR in testing). At least this would show some robustness of their approach to the already existing distribution of missing data. \n\n- Supporting Section 2.4 in Section 4.1: Section 4.1 offers the opportunity for the authors to support their proposal for NMAR data. Namely, they could compare (i) treating all the existing observations as a regular observation (and potentially ampute+impute) (ii) treating the NMAR observations separately. \n\n- I think that the comparison Knockout vs. Knockout* helps capture the benefit of the OOR \\bar{x} in the overall approach vs mean impute. Analogously, I think it would be good to implement and report the Common baseline with OOR \\bar{x} instead of mean impute, to better illustrate the benefit of the second ingredient of the approach, namely synthetically amputing at training time. \n\n- Appendix p. 19 “the dataset can be seen in 9.” -> the dataset can be seen in Table 9.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "d1sGdkgL2y", "reviewer": "Reviewer_2RrG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n+ The paper is generally well-written and easy to follow.\n+ The authors provide a strong theoretical foundation, supported by detailed mathematical analysis.\n+ The comprehensive evaluation covers low- and high-dimensional datasets, synthetic and real-world tasks, and varied domains (e.g., clinical data, segmentation), showcasing the method’s versatility.\n\nWeakness:\n- The idea of randomly dropping inputs during training is not new; many related works follow the same intuition.\n- The empirical validation does not include vision-language datasets. A key assumption of using rich sources to train the model is that different modalities provide complementary information. Image and text serve as a good example of this, so including results on a vision-language dataset would further strengthen the validity of the proposed method.\n- In scenarios with limited training data, how well does Knockout perform compared to mean or mode imputation?\n - Considering the recent advances in leveraging rich sources to train multimodal models, one key question is not discussed: Do the missing inputs (modality) challenge continue to be a critical issue for more recent multimodal models, such as LLaVA or VILA, given advancements in multimodal learning?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces “knockout ” to improve model robustness against missing inputs. The key idea is to randomly “drop out” input features during training. The author provides theoretical justification from the implicit marginalization perspective and demonstrates its effectiveness across various settings, including synthetic data, clinical predictions, noisy label learning, tumor segmentation, and multi-modal classification tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "+ Consider including empirical evaluations on datasets where the inputs are much diverse, such as the Hateful Memes dataset [1]  or Food-101 [2].\n+ Add discussion or experimental comparisons regarding Knockout’s performance relative to baseline imputation methods in low-data scenarios.\n+ Expand the manuscript to address how Knockout relates to the broader issue of missing inputs in recent multimodal foundation models.\n\n*References:*\n\n[1] Kiela, Douwe, et al. \"The hateful memes challenge: Detecting hate speech in multimodal memes.\" Advances in neural information processing systems 33 (2020): 2611-2624.\n\n[2]  Wang, Xin, et al. \"Recipe recognition with large multimodal food dataset.\" 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW). IEEE, 2015.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)