Generative Imputation and Stochastic Prediction
Authors: Mohammad Kachuee
Kimmo Kärkkäinen
Orpaz Goldstein
Sajad Darabi
Majid Sarrafzadeh
Venue:  (2020)

Abstract: In many machine learning applications, we are faced with incomplete datasets. In the literature, missing data imputation techniques have been mostly concerned with filling missing values. However, the existence of missing values is synonymous with uncertainties not only over the distribution of missing values but also over target class assignments that require careful consideration. In this paper, we propose a simple and effective method for imputing missing features and estimating the distribution of target assignments given incomplete data. In order to make imputations, we train a simple and effective generator network to generate imputations that a discriminator network is tasked to distinguish. Following this, a predictor network is trained using the imputed samples from the generator network to capture the classification uncertainties and make predictions accordingly. The proposed method is evaluated on CIFAR-10 image dataset as well as three real-world tabular classification datasets, under different missingness rates and structures. Our experimental results show the effectiveness of the proposed method in generating imputations as well as providing estimates for the class uncertainties in a classification task when faced with missing values.

Under review as a conference paper at ICLR 2020
GENERATIVE IMPUTATION AND STOCHASTIC PREDICTION
Anonymous authors
Paper under double-blind review
ABSTRACT
In many machine learning applications. we are faced with incomplete datasets. In the literature, missing data imputation techniques have been mostly concerned with filling missing values. However, the existence of missing values is synonymous with uncertainties not only over the distribution of missing values but also over target class assignments that require careful consideration. In this paper, we pro- pose simple and effective method for imputing missing features and estimating the distribution of target assignments given incomplete data. In order to make imputations, we train a simple and effective generator network to generate im- putations that discriminator network is tasked to distinguish. Following this, a predictor network is trained using the imputed samples from the generator net- work to capture the classification uncertainties and make predictions accordingly. The proposed method is evaluated on CIFAR-10 image dataset as well as three real-world tabular classification datasets, under different missingness rates and structures. Our experimental results show the effectiveness of the proposed method in generating imputations well providing estimates for the class uncertainties in a classification task when faced with missing values.
INTRODUCTION
While large body of the machine learning literature is built upon the assumption of having access to complete datasets, in many real- world problems only incomplete datasets are available. The existence of missing values can be due to many different causes such as human subjects not adhering to certain questions or features not being collected frequently due to financial or experimental limitations, sensors failures, and so forth. Data imputation techniques have been suggested as a solution to bridge this gap in the literature by replacing missing values with observed values.
Missing data imputation approaches can be categorized into single and multiple imputation methods. Single imputation methods try to replace each missing value with plausible value that is the best fit given the value of other correlated features and knowledge extracted from the dataset (Hastie et al. 1999. Anderson 1957). While these methods are easy to implement and use in practice, imputed values may induce bias by eliminating less likely but important values. Also, these methods do not suggest a way measure what extent the imputed values representative of the missing values (Little& & Rubin 2019).
Multiple imputation (MI) techniques, as suggested by the name, try to use multiple imputed values to impute each missing value. The result would be having a set of imputed datasets that enables measuring how consistent and statistically significant are the results of the experiments (Rubin) 1976). While MI offers interesting statistical insights about the reliability of analysis on incomplete data, the insight is imprecise as it is mainly concerned about the population of data samples rather than individual instances. Specifically, MI methods reason about the statistical properties on limited number of imputed datasets (less than 10 most practical implementations) on the population of samples within the dataset (Schafer & Graham 2002 Murray et al.) 2018).
The existence of missing values is synonymous with having uncertainty over these values that requires careful consideration. In many real-world applications, we are dealing with supervised problems that demand modeling and prediction based on incomplete data. Take for instance, prediction of class assignments given an image in which large portion of the frame is missing. In such scenario, based on observed frame parts, there might be multiple probable class assignments each having a different likelihood, Here, we not only interested in imputing missing values measuring how

---

Under review as a conference paper at ICLR 2020
robust our imputations are, but also it is highly desirable to measure the impact of missing values on the prediction outcome for each instance.
In this paper, propose the idea of Generative Imputation and Stochastic Prediction (GI) as a novel approach to impute missing values and to measure class uncertainties arising from the distribution of missing values. The suggested approach is based neural networks trained using adversarial objective function. Additionally, a predictor is trained on the generated samples from the imputer network which is able to reflect the impact of uncertainties over missing values. This enables measuring different prediction outcomes and certainties for each specific instance. We evaluate the effectiveness of the proposed method on different incomplete image and tabular datasets under various missingness structures.
2 RELATED WORK
One of the simplest traditional methods for handling missing values includes imputing the occurrences of missing values with constant values such as zeros using mean values. To enhance the accuracy of such imputations, alternatives such as k-nearest neighbors (KNN) (Hastie et al. 1999) and maximum likelihood estimation (MLE) (Anderson 1957) have been suggested to estimate values to be used given an observed context. While these methods are easy to implement and analyze, they often fail to capture the complex feature dependencies well structures present in many problems.
Rubin 1976) suggested a categorization for missingness mechanisms: missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR). Under the assumption of MAR, the authors suggested multiple imputation (MI) as a stochastic imputation method. Here, instead of imputing missing values using a single value, several values are sampled to represent the distribution over the missing value. MI generates few imputed complete datasets that are then used independently in statistical modeling (Schafer Graham 2002 Little Rubin 2019). Usually, the final goal of MI is to measure the robustness of the final statistical analysis amongst the imputed datasets. In other words, it measures the quality of imputations and the statistical significance of analysis on the imputed data. It should be noted that the number of imputations used in MI is usually very limited. Also, often strong simplifying assumptions are made in modeling the data distribution (e.g., multi-variate Gaussian or Student's distribution) which limit the applicability of this method (Schafer & Graham 2002 Murray et al.) 2018).
More recently, autoencoder architectures have been suggested as powerful density estimators capable of capturing complex distributions. Perhaps, denoising autoencoders (DAE) (Vincent et al. 2008) are one of the most intuitive approaches in which neural network is trained to reconstruct and denoise its input. Following a more probabilistic perspective, variational autoencoders (VAB) (Kingma & Welling 2013) try learn the data generating distribution via a latent representation. Specifically, conditional variational autoencoders (CVAE) (Sohn et al. (2015) can be used to sample missing values Mattaanee,MMteiii conditioned on observed values. For instance, Mattei Frellsen (2018) suggested method based on deep latent variable models and importance sampling that offers a tighter likelihood bound compared to the standard VAE bound. While these methods are powerful generative models applicable to missing data imputation, often samples generated using autoencoders are biased toward the mode of the distribution (c.g., resulting in blurry images, for vision tasks) (Goodfellow et al. 2014 Dumoulin ct al. 2016).
Recently, due to the success of generative adversarial networks (GAN), there has been great attention
toward applying them to impute missing values. For instance, Yoon et al. (2018) suggested an imputation method based on adversarial and reconstruction loss terms, Li et al. idea of using separate generator and discriminator networks to learn the missing data structure and data distribution. These methods have been quite successful and are able to present the -tate-of-the-art results. Though it should be noted that often the presence of additional loss terms may bias the generated samples toward the mode of the distribution being modeled. Also, these methods are often complicated to be applied in practical setups by practitioners. For instance, Yoon et al. (2018) requires setting hyperparameters to adjust the influence of an MSE loss term as well as the rate of
2019) introduced the
We plan to include link to the source code and GitHub page related to this paper in the camera-ready version.
2

---

Under review as a conference paper at ICLR 2020
X;
ki
Xi
D(2,)
k
Figure 1: Block diagram of the proposed adversarial imputation method. represents the blending function of (T), and L is the adversarial loss function of (
discriminator hint vectors. Also another example, Li et al. (2019) uses three generators and three discriminators for the final imputer architecture.
From the perspective of supervised analysis. imputation and handling missing values usually considered as a preprocessing step, A few exceptions exist such as Bayesian models and decision trees that permit direct analysis incomplete data (Nielsen & Jensen 2009 Zhang al. 2005). Note that while certain Bayesian methods such as probabilistic Bayesian networks allow handling of missing values unobserved variables. However, given incomplete training dataset and without any known causal structure as priori, learning such models is very challenging problem with the complexity of least NP-complete learn the network architecture in addition an iterative EM optimization to learn model parameters (Darwiche) 2009 Ncapolitan et al., (204)). We argue that the simplistic approach of imputing missing values as a preprocessing step discards uncertainties that exist in original incomplete data samples. Instead, there is need for methods that reflect these uncertainties on the final predicted target distribution. This work suggests the idea of training predictor on different imputed samples to capture the uncertainties over class assignments. Compared to MI, the suggested method interleaves imputation and training a downstream prediction model, enabling to estimate classification uncertainties for each instance.
PROPOSED METHOD
3.1 PROBLEM DEFINITION
In this paper, make the general assumption of having access incomplete dataset 'D consisting of a set feature vector, mask vector, and target class pairs (æ,, k, 34). For each feature vector, a; E Rd, only subset of the features is available. The mask vector k; E {0, 1] is used to indicate available features and missing features by ones and zeros, respectively. Here, to represent features as fixed-wiid width vectors, arbitrary (or NaN) values are used to fill missing values. Also, for convenience, we often use xo and simie to refer to the set of observed and missing features for the feature vector 2t;
We define our objective in two steps: (i) Imputing missing values via sampling from the conditional distribution of missing features given observed features i.e., P( laqbe). (ti) Estimating the distribution of target classes given the observed features and the distribution of missing features i.e., P(y| x mmmsss) ). For the first part, we are interested sampling from the conditional distribution rather than finding the mode of the distribution as the most probable imputation. Similarly, for the second part, are interested in obtaining a distribution over the possible target assignments and the confidence of each class rather than maximum likelihood class assignments.
3.2 GENERATIVE IMPUTATION
To generate samples from the distribution of missing features conditioned on the observed features, we follow the idea first suggested by Yoon et al. (2018). In this paradigm, a generator network is responsible for generating imputations while discriminator is trying distinguish imputed features from observed features (see Figure [
Specifically, the generator function G(æ,, k;, 2) Rd generates imputed feature vector, based on observed features, the corresponding mask, and a Gaussian noise vector (z). Note that, in order to achieve the final imputed vector, R; blend (or, merge) the output of the generator with the input
3

---

Under review as a conference paper at ICLR 2020
features to replace generated values with the exact values of observed features:
if ki.j G( 2) if k..j 0
D.j
where Vi refers to j'th feature of sample i. Also, note that by sampling = multiple times, we can obtain different imputation samples from the conditional distribution indicated by zi where the sample number.
discriminator network, D is trained to distinguish real and imputed features by generating a predicted softmax mask output, k;. Here a binary cross-entropy loss per mask element is used as the adversarial objective function:
max min L(G, = D... [k" (k) + - (1-k)" log(1 - k)]-
(2)
G
The intuition behind this adversarial loss function is that given generator function which captures the data distribution successfully, the discriminator would not be able to distinguish the parts of the feature vector that were originally missing.
Compared to Yoon et al. 2018), the objective function of 2 does not have an MSE loss term. Instead, we use recent advances in GAN stabilization and training improve the training process (see Section 3.4). While it is quite prevalent in the adversarial learning literature to use additional loss terms such squared error (MSE) to enhance the quality of generated samples, decided to keep solution as simple as possible. Additionally, in our experiments, we provide supporting evidence that this simple loss function enables to sample from the conditional distribution and prevents biased inclinations toward distribution modes.
3.3 STOCHASTIC PREDICTION
To capture the distribution target classes given incomplete data, we suggest the idea of stochastic prediction. As indicated in the previous section, the generator can be used to sample from the conditional distribution. Here, predictor is trained based on the imputed samples to predict class assignments and to calculate the confidence of these assignments. For instance, for specific test sample hand, a certain missing feature a strong indicator of the target class, we would like to observe the impact of different imputations for that feature the final hypothesis.
Formally, we are interested finding the certainty of class assignments given observed features:
V Plyl (
Here, is vector where each element is representing certain class. Rewriting B as marginal we have:
v / P(z P Pyllæ,"" obs x xiiiss) dæ:
(4)
Approximating the integration using a summation, given enough samples, can be estimated by:
V F= N Σ P(yla a.misa),
1
(5)
where Semses are samples taken from the conditional distribution of missing features given observed ones. We use the suggested generative imputation method to generate samples required for this approximation. Rewriting (T) using Hadamard product and function of the noise vector:
k, G x; (1 k.) G( k.,
Assuming that predictor, Fo. is available which predicts class assignments for a complete feature vector, V can be estimated as:
[()(( 1 EN
Algorithm I presents the suggested algorithm for training the predictor. It consists of taking samples from the incomplete dataset, then imputing them using our generator network, and using the imputed
4

---

Under review as a conference paper at ICLR 2020
Algorithm 1: Training the predictor.
Algorithm 2: Estimating target distributions.
Input: G (trained imputer). D (dataset)
Input: Fg (trained predictor), (x, k) (test sample), N (ensemble samples)
Output: Fo (trained predictor)
foreach Training Epoch do
Output: v (distribution over target classes) V zeros E R#ccossee
foreach (I,, ko y:) in D do
2 (0,
foreach Ensemble Sample I to N do
& &
O (R,)
N(O.))
X f l o x (1- G( k. 2)
loss t L(yi, y )
good + (x
- arrggaz (amaazyyred (y
Backpropagate loss
4 t V
Update Fo
samples update the predictor. Note that, on each epoch and for each sample, the generator generates a new sample from the conditional distribution. Intuitively, it means that the predictor observes and learns to operate under different imputations for given sample. This is different from approaches such as multiple imputation where several predictors are trained on different imputed versions of dataset.
Algorithm 2 presents the suggested algorithm for making predictions and estimating target distribu- tions given trained predictor model. Here, a sample imputed N times and inference on this set results in ensemble of predictions over different imputations. The output of this algorithm can be interpreted as distribution over the confidence of class assignments given a partially observed test sample. The following claims justify the validity of Algorithm and Algorithm 2
Claim 1. (Generalization of the predictor). If we assume imputed i,s are samples from the underlying feature distribution, then the assigned training set labels can be modeled as labels generated from noisy labeling process.
Claim 1 permits the analysis of the generalization and convergence for the predictor trained using Algorithm based on current literature in training models with noisy labels (Natarajan et al.] 2013
RRed et al. 2014 Chen et al. 2019). From the analysis provided by Chen et al. (2019), test accuracy
in asymmetric label noise conditions a quadratic function of the label noise:
P(y 73) (1 c)² +2
where Ys is underlying true label for the imputed feature vector (ε)), and y, is the label provided by the incomplete dataset. In (8). label noise ratio, e, represents the probability of the label transition from certain target class to another:
6-1- P(ū = jlys = j).
In practice, is determined by the problem-specificc underlying data distribution as well as the distribution of missing values.
Justification for claim is straightforward, assume that " ... Di N are underlying true labels for each of ".. St .,,,,..,.""...... R. During training, for any imputed sample in { ... 2, }. we use the dataset provided label, to calculate the loss and to update model parameters. In the case that any of T. ( } is different from y. the loss term corresponding to that term would be calculated using a wrong label. Here, we consider the average impact on gradients for batches of samples rather than individual cases, the overall impact on training would be very similar to the case of training using noisy labels.
Claim 2. (Approximation of the target distribution). If we assume:
(i) imputed i,s are valid samples from the underlying feature distribution, (ii) good predictor can be trained using the incomplete data, (ii)) enough samples are used and the Monte Carlo estimator is unbiased, then the target distribution, V. can be estimated accurately.
This claim supports Algorithm 2 that is suggested to estimate the target distribution given a partially observed feature vector.
5

---

Under review as a conference paper at ICLR 2020
The first assumption is consistent with the theoretical analysis of generative adversarial networks that they can converge to the true underlying distribution (Arora et al. 2018 Liu et al., 2017). The second assumption is supported by Claim 1. Regarding the last assumption, each sample requires one forward computation of the generator and predictor networks which, based on the scalability of current network architectures, usually permits thousands of samples to be taken at reasonable computational cost.
3.4 IMPLEMENTATION DETAILS
As we conduct experiments on image and tabular datasets, we use different architectures for each. For image datasets, we used generator and discriminator architectures similar to the ones suggested by Waag et al. (2018). However, we improved these architectures using self-attention layers (Zhang et al. (2018). It should be noted that, while ZZang et al.) (2018) suggests using single self-attention layer in the middle of the network, we observed consistent improvements by inserting multiple self-attention layers before each residual block within the network. Furthermore, as input to the generator, we concatenate input image, mask, and a random z frame along the channels dimension and use as input. For tabular datasets, we use a simple 4 layer network consisting of fully- colly-connected and batch-norm layers. Also, the input to the generator is the concatenation of a feature vector, mask vector, and a z vector of size g of the input. For all experiments, we use an ensemble size (N) equal to 128.
We used Adam (Kingma Ba 2014) for model optimization. Two time-scale update rule (TTUR) (Heusel et al.) (017)) was used to balance training the generator and discriminator net- works. We explored best TTUR learning-rate settings from the set of [0.001, 0.0005, 0.0001, 0.00005). Here, Adam parameters 33 and 83 are set to 0.5 and 0.999, respectively. Also, spectral normalization was used to stabilize both the generator and discriminator network in our experiments with image data (Miyato et al. 2018). For the predictor network, we used the default Adam settings as suggested by Kingma & Ba 2014). In all training procedures, we decay learning rate by a factor of 5 after reaching a plateau. For all experiments, we batch size of 64. Based on experiments, we found that pretraining the discriminator while fixing the generator network for the first 5% of the training epochs helps the stability of training.
Further detail on exact architectures, experiments, software dependencies, etc. as well as ablation studies is provided in the appendices.
EXPERIMENTS
4.1 DATASETS
To evaluate the proposed method we use CIFAR-10 (Krizhevsky & Hinton 2009) as an image
classification dataset as well as three non-image datasets: UCI Landsat (Dua Graff 2017 2
MIT-BIH arrhythmia (Moody & Mark 2001), and Diabetes classification (Kachuee et al. 2019
CIFAR-IO dataset consists of 60,000 32x32 images from 10 different classes. For this task, we use train and test sets as provided by the dataset. As a preprocessing step, we normalize pixel values to the range of [0,1] and subtract the mean image. The only data augmentation we use for this task is to randomly flip training images for each batch.
UCI Landsat consists of 6435 samples of 36 features from 6 different categories. We follow the same train and test split as provided by the dataset. MIT-BIH dataset consists of annotated heartbeat signals from which we used the preprocessed version available o consisting of 92062 samples of 5 different arrhythmia classes Diabetes dataset a real-world health dataset of 92,062 samples and 45 features from different categories such as questionnaire, demographics, medical examination, and lab results. The objective to classify between three different diabetes conditions i.e., normal, pre-diabetes, and diabetes. As MIT-BIH and Diabetes datasets do not provide explicit train and test sets, we randomly select 80% of samples as a training set and the rest as test set. To preprocess our tabular datasets, statistical and unity based normalization are used to balance the variance of
hhtpaess/accciee.......eeee//ees http:///ttttttoooaaaaaaaaaaattee (landsattSatellitte)
https://www.kagg111 com htt.://gggge...ooaaaaa.aa.aefeee htw.://eegg....ooohaaaaaaaa////a

---

Under review as a conference paper at ICLR 2020
different features and center them around zero. Also, while different encoding and representation methods are suggested in the literature to handle categorical features (Jang et al. 2016 Nazabal et al. 2018), in this paper, take the simple approach of encoding categorical vanables using one-hot representation and smoothing them by adding Gaussian noise with zero mean and variance equal to 5% of feature variances. In our experiments, we observed reasonable performance using the suggested simple smoothing: however, more advanced encoding methods are also applicable in this setup and can be applied to enhance the performances even further.
4.2 MISSINGNESS MECHANISMS
In experiments, consider MCAR uniform and MCAR rectangular missingness structures. In MCAR uniform, each feature of each sample is missing based on a Bernoulli distribution with certain missingness probability (i.e., missing rate) independent of other features. In addition to the case of uniform missingness, for image tasks, we use rectangular missingness/observalii structure where rectangular regions of dataset images are missing/observed. To control the rate of missingness and decide on the regions that are missing for each case, we use latent beta distribution that samples rectangular region's width and height such that the average missing rate is maintained. For missing rates less than 50% we make the assumption of having random rectangular region to be missing, whereas for missing rates more than 50% we assume that only a random rectangular region is observed and the rest of the image is missing.
We would like to note that while the suggested solution in this paper readily compatible with MAR structures, in our experiments, to simplify the presentation of results and to have fair comparison with other work that does not support the MAR assumption, we limited the scope of our experiments to MCAR. Furthermore, to simulate incomplete datasets and to make sure the same features missing without explicitly storing masks, we use hashed feature vectors to seed random number generators used to sample missing features. More detail is provided in Appendix C
4.3 EVALUATION MEASURES
Fréchet inception distance (FID) (Heusel et al. 2017) score is used to measure the quality of missing data imputation in experiments with images We also considered using root means squared error (RMSE); however, we decided not to use this measure as we observed inconsistent behavior using RMSE our comparisons as RMSE favors methods that show less variance rather than realistic and sharp samples from the distribution. Also, for each dataset and each missingness scenario, report top-1 classification accuracy based on the majority vote estimated using Algorithm [ Another measure that
in this paper is the comparison between the estimated target certainties and average accuracies achieved for each confidence assignment. We run each experiment multiple times: 4 times for CIFAR-I0 and times for tabular datasets. We report the mean and standard deviation of results for each case.
We compare our results with MisGAN et al. 2019) and GAIN (Yoon et al. 2018 as the state of the art imputation algorithms based on GANS as well as basic denoising autoencoder (DAE) (Vincent imputeate al. 2008) and multiple imputation by chained equations (MICE) (Buuren & Groothuis- Oudshoo 2010) as baselines. For experiments using MisGAN, we used the same architectures and hyper- parameters as suggested by the MisGAN a The only modification was to adapt the last generator layer to generate images with resolutions as we use. Regarding GAIN, we used the same network architecture as our implementation of GI and hyper-parametes as used by the GAIN authors" In the DAE implementation, due to the incomplete data assumption, only observed features appear in the loss function, ignoring reconstruction terms corresponding to missing features. Due to scalability issues, we were only able to use MICE for the smaller non-image datasets. For these methods, to train and evaluate classifiers, we use predictors trained on imputed datasets rather than the stochastic predictor suggested in Algorithm
7

---

Under review as a conference paper at ICLR 2020
CIFAR-10 MCAR Uniform
CIT 10 MCAR Rect.
61
250
ME24AN GAN
MSGAN
GAIN
DA2
DAE
200
150
52-
50-
Missing Rate (%)
Missing (56)
(a)
(b)
Figure 2: Comparison of FID scores on CIFAR-10 dataset for (a) uniform and (b) rectangular missingness. Lower FID score is better. In many cases, variance values are very small and only observable by magnifying the figures.
4.4 RESULTS
Figure presents the comparison of FID scores on the CIFAR-10 dataset different missing rates for uniform and rectangular missingness. As it can be inferred from these plots, GI outperforms other alternatives in all cases. Also, can be seen that GAIN is able provide more reasonable results for uniform missing data structure compared to MisGAN which is mainly effective in the rectangular missing data structure. One possible explanation for this behavior might be the fact that GAIN has an MSE loss term acting similar an autoencoder loss smoothing noisy missing pixels. On the other hand, MisGAN tries to explicitly model missingness structure and is more successful in capturing a more structured missingness such the case of rectangular structure. Table [ provides a comparison between the top-1 I classification accuracy achieved using each method at different missing rates and structures. From this table, GI outperforms other work by achieving the best results in out c
Trrvierrov
Table presents comparison of classification accuracies for Landsat, MIT-BIH and Diabetes datasets at different missing rates. In the Landsat benchmark, GI outperforms other work in all cases. Regarding the MIT-BIH experiemts, GI outperforms other work for missing while achieving similar accuracies to GAIN for lower missing rates. In the diabetes classification task, GI appears be most effective imputing missing rates more than 20%.
more than 30%
Figure shows comparison of accuracy versus certainty plots for GI, MisGAN, and GAIN on Landsat dataset at the missing rate of 40%. To generate these figures trained each imputation method and then used Algorithm| I to train predictors on imputed samples. Finally, Algorithm 2 used to measure the average accuracy different prediction confidence levels based on sample of 128 imputations for each test example. As it can be seen from the plots, GI provides results closest to the ideal case of having average confidence values equal to average accuracies.
4.5 VISUALIZATION USING SYNTHESIZED DATA
In order provide further insight into the operation of GI and how imputations can potentially influence the outcomes of predictions, we conduct experiments on synthesized dataset. The original underlying data distribution is generated by sampling 5000 samples from 4 Gaussians of standard deviation 0). centered on the vertices of unit square. We assign two different classes to each cluster
hutterr//gaaab.cco//eetttt/rr/rr fid adapted to measure the FID scores.
https: "ittatr///dtttal.om//teee//////
https://gi thub. cwm httph//.cccomoooooo2222////////
An earlier version of this paper reported results that are different from the current manuscript. The current version using the stochastic predictor exclusively on the suggested imputation method and trained using more precise hyper-parameter settings.
8

---

Under review as a conference paper at ICLR 2020
Table 1: Top-1 CIFAR-IO classification accuracy for different missing rates and structures.
Accuracy at Missing Rate (%)
MCAR Uniform
MCAR Rect.
Method
20%
40%
60%
20%
40%
60%
GI
89.5 (4:0.45)
87.1 (+0.54)
80.3 (=0.26)
84.0 (#0.03)
76.9 (#0.03)
66.1 (#0.16)
MisGAN
86.5 (1.0311)
83.7 (1.0.0)
78.7 (10.266)
82.9 (1:044)
75.6 (1.0.20)
65.0 (1:0.31)
GAIN
88.7 (1.0.45)
86.0 (10086)
81.8 (1:0.03)
81.7 (1:0033
73.6 (100335
58.4 (d11.66)
DAE
88.0 (1.0.22)
84.0 (1.0.50)
79.8 (110.71)
83.3 (1:064)
75.5 (10.44)
63.8 (1:0.24)
Mean
85,7 (#:0.02)
83.4 (+0.38)
79.2 (=0.16)
82.7 (#0.15)
75.3 (#0.16)
64.0 (#0.332)
Table 2: Comparison of classification accuracies for Landsat, MIT-BIH, and Diabetes datasets at different missing rates.
Accuracy at Missing Rate (% 4
Dataset
Method
10%
20%
30% %
40%
GI
89.9 (=0.36)
89.6 (#0036)
89.0 (20.03)
88.0 (#:0.22)
MisGAN
87.2 (20.01)
85.7 (::0.19)
84.0 (:00661)
82.9 (d:0.75)
Landsat (uua & Graff. 2017
GAIN
89.7 (20.42)
89.4 (:00566
88.4 (10.711)
87.7 (400110
DAE
89.4 (20.10)
88.6 (:0.54)
87.5 (2:0.14)
86.6 (#:0.21)
MICE
89.5 (#0.16)
89.3 (#0.10)
88. (#0.49)
87.5 (#00.03)
GI
98.5 (20.02)
98.4 (::0.03)
98.2 (10.07)
97.7 (::0.03)
MisGAN
97.8 (20.13)
97.4 (#00.07
96.7 (20007)
96.2 (#:0.09)
MIT-BIH Moody & Mark, 2001
GAIN
98.5 (#0.02)
98.4 (#0.000 06)
98.0 (#0.09)
97.5 (#00.18)
DAE
98.4 (20.02)
98.2 (#00.11)
97.9 (200009)
97.4 (#:0.02)
MICE
98.4 (=0.01)
98.3 (+0.00))
98.1 (20.00)
97.5 (#:0.12)
GI
89.6 (2:0.13)
89.0 (::0.03)
88.2 (::062)
86.8 (::0.38)
MisGAN
89.7 (20.01)
88.9 (20.30)
87.6 (20.00)
86.4 (#:0.68)
Diabetes Kachuuee et al. 2019
GAIN
89.2 (=0.09)
88.3 (#0.02)
86.9 (20.09)
83.8 (#1144)
DAE
89.3 (#0.055
88.2 (::0.19)
86.9 (#0.09)
84.8 (1:0.03)
MICE
89.8 (10.08)
88.8 (10.001)
88.0 (10.08)
86.1 (40.00)
"Baseline accuracies for complete datascts (zero missing rate) are equal 90.9%, 98.6%, and 90.7% for Landsat, MIT-BIH and Diabetes, respectively.
such that diagonal vertices are of the same class (see Figure 4r. classes are represented with colors). From this underlying distribution, we make an incomplete dataset with 50% of values missing.
The incomplete synthesized dataset is used train GI and other imputation methods. We take random test sample in which the second feature has a value of about 0. and the other feature is missing, Ideally, in the imputation phase, we would like to sample from the condition distribution ie.. P(z, |x2 = 0.1) (see Figure Here, in the prediction phase, an ideal method would decide on not making confident classification and report the uncertainty. Note that solely observing the value of 0.1 for the second feature does not provide any useful evidence for the prediction. ff provide samples and classification results for GI, MisGAN, GAIN, and DAE, As it can be inferred from these figures, GI generates reasonable samples from the conditional distribution and also reflects this uncertainty over the prediction. On the other hand MisGAN, probably due to its complexity of using three different generators and discriminator pairs, is suffering from mode collapse and is unable to generate samples from the other class, resulting in over- aver-confident assignments. GAIN, perhaps due to the MSE loss terms, is inclined towards the mean of the conditional distribution at the origin.

---

Under review as a conference paper at ICLR 2020
Ideal
Ideal
Ideal
Accuracy
Accuracy
Accuracy
0.00
0.25
0.25 Confidence
1.00
Confidence
Confidence
(a)
(c)
Figure 3: Accuracy versus certainty plots for (a) GI, (b) MisGAN, and (c) GAIN on Landsat dataset the missing rate of 40%.
DAE, expected, due to its MSE loss term, only captures the expected value of the distribution mean hence reducing the MSE error and generates over-smoothed imputations.
Original P()).x2)
Gl -- |xx = 0.1)
GAIN P(x] 0.1)
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.50
X 0.50
0.25
0.25
0:00 0
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
0.25
0.50
0.75
1.00
X1
(e)
Original PXg 0.1)
MisGan X2 0.1)
P(x, (X2 0.1)
0.50
8 0.50
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
X1
X1
(b)
(d)
(f)
Figure Evaluation using synthesized data: (a) samples from the underlying distribution, (b) samples from the conditional underlying distribution, (c-f) samples from the conditional distribution generate by GI. MisGAN, GAIN, and DAE.
CONCLUSION
In this paper, we proposed novel method to generate imputations and measure uncertainties over target class assignments based on incomplete feature vectors. We evaluated the effectiveness of the suggested approach on image and tabular data via using different measures such as FID distance, classification accuracy, and confidence versus accuracy plots. According to the experiments, the proposed method not only can generate accurate imputations but also is able to model prediction uncertainties arising from missing values. The proposed method is applicable many real-world applications where only an incomplete dataset is available, and modeling classification uncertainties necessity.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Theodore W Anderson. Maximum likelihood estimates for a multivariate normal distribution when some observations are missing. Journal of the american Statistical Association, 52(2788:200-203, 1957.
Sanjeev Arora, Andrej Risteski, and Yi Zhang. Do gans learn the distribution? some theory and empirics. 2018.
S van Buuren and Karin Phoothuis-Oudshoom mice: Multivariate imputation by chained equations in r. Journal of statistical software, pp. 1-68, 2010
Pengfei Chen, Benben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. arXiv preprint arXiv: 1990505044) 2019.
Adnan Darwiche. Modeling and reasoning with Bayesian networks. Cambridge university press, 2009.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http://archioe 1cs. www..ce.ue.........
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, and Aaron Courville. Adversarially learned inference. arXiv preprint orXiv:1606.000 00704, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems, pp. 2672-2680, 2014.
Trevor Hastie, Robert Tibshirani, Gavin Sherlock, Michael Fisen, Patrick Brown, and David Botstein. Imputing missing data for gene expression arrays, 1999.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by time-scale update rule converge a local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax arXiv preprint arXiv:1611.01 144, 2016.
Mohammad Kachuee, Kimmo Karkkainen, Orpaz Goldstein, Davina Zamanzadeh, and Majid Sar-
rafzadeh. Nutrition and health data for cost-sensitive learning. arXiv preprint 102.02190..07102. 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. 1rXiv:1312.6114,22 2013.
arXiv preprint
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Steven Cheng- Xian Li, Bo Jiang, and Benjamin Marlin. Learning from incomplete data with generative adversarial networks. In International Conference on Learning Representations, 2019. UtL.hhttss://peereviii
net/forumm weeiiidd=ssdseeee
Roderick JA Little and Donald B Rubin. Statistical analysis with missing data, volume 793. Wiley, 2019.
Shuang Liu, Olivier Bousquet, and Kamalika Chaudhuri. Approximation and convergence properties of generative adversarial learning. In Advances in Neural Information Processing Systems, pp. 5545-5553, 2017.
Flexre-Ae Mattei and Jes Frellsen. missiwae: Deep generative modelling and imputation of incomplete data. arXiv preprint /812,0033,, 2018.

---

Under review as a conference paper at ICLR 2020
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957, 2018.
George B Moody and Roger G Mark. The impact of the mit-bih arrhythmia database. IEEE Engineering in Medicine and Biology Magazine, 20(3):45-50, 2001.
Jared S Murray et al. Multiple imputation: a review of practical and theoretical findings. Statistical Sciiece, 33(2):142-155) 159, 2018.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in neural information processing systems, pp. 1196-1204, 2013.
Alfredo Nazabal, Pablo M Olmos, Zoubin Ghahramani, and Isabel Valera. Handling incomplete heterogeneous data using vaes. arXiv preprini arXiv: arXiv: 1807.0363, 2018.
Richard Neapolitan al. Learning bayesian networks, volume 38. Pearson Prentice Hall Upper Saddle River, NJ, 2004.
Thomas Dyhre Nielsen and Finn Verner Jensen. Bayesian networks and decision graphs. Springer Science & Business Media, 2009.
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks noisy labels with bootstrapping. arXiv preprint arXiv:1412.6596, 2014.
Donald B Rubin. Inference and missing data. Biometrika, 63(3):581-592, 1976.
Joseph L Schafer and John W Graham. Missing data: our view of the state of the art. Psychological methods, 7(2): 2002.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Advances in neural information processing systems, pp. 3483-3491, 2015.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Fierre-Antoing Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103. ACM, 2008.
Fing-Chun Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8798-8888 2018.
Jinsung Yoon, James Jordon, and Mihaela Van Der Schaar. Gain: Missing data imputation using generative adversarial nets. arXiv preprint arXiv:1806.02922 2018.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. arXiv preprint arXiv: N00.:8805.0814 2018.
Shichao Zhang, Zhenxing Qin, Charles X Ling, and Shengli Sheng. missing is useful": missing values in cost-sensitive decision trees. IEEE transactions knowledge and data engineering. 17 (12):1689-1193, 2005.
12

---

Under review as a conference paper at ICLR 2020
A IMPLEMENTATION AND EXPERIMENTS
Table presents the list of software dependencies and versions used in our implementation. produce results related to this paper. used workstation with 4 NVIDIA GeForce RTX-2080TI GPUs, a core Intel Core 79279 processor, and 128 GB memory. Each experiment took between about 4 hours to 48 hours, based the task and method being tested.
Table Software dependencies.
Dependency
Version
python
3.7.1
pyterch
1.0.0
cuda100
1.0
ipython
7.2.0
jupyter
1.0.0
numpy
1.15.4
pandas
0.24.1
scikit-learn
0.20.1
scipy
1.1.0
torchvision
0.2.1
tqdm
4.28.1
matplotlib
3.0.1
B NETWORK ARCHITECTURES
Table I shows the exact architectures used in this paper. show each layer or block we used following notation. CxSyPz-t represents 2-d convolution layer of kernel size stride y, padding 2, and number output channels t followed by ReLU activation. tt represents a self-attention layer similar to Zhang et al at" R-x represents residual block consisting of two 2-d convolutions with kernel size (padding size 1), batch normalization, and ReLU activation. CTxSyPz z-t. t the convolution transpose corresponding to CxSyPz-te z-t. FC-x is representing linear fully-connected layer of x output neurons with biases. We use spectral normalization as suggested by Miyato et all 10 for all convolutional layers in both generator and discriminator networks.
Table 4: Network architectures used in our experiments.
Dataset
Gennertorlliserimnnater Architecture
Predictor Architecture
C781F3 64, C352P1 128, Attn, R-128,
CIFAR-10
Attn, R-128, Aste, R-12a, Attm,
ResNet-18
R-128, cT352P1-128, CT751P3-3, Tanh/Siqmoid
FC-64, 3igmoid, BNorm, FC-64, Sigmoid, RNorm,
FC-64, Retül RNorm, FC-64,
Landsat
FC-64, Sigmcid, BNorm, FC-36, fanh/Sigmoid
Relil BNorm, FC-6, Softmax
FC-1860, ReLU, BNerm, FC-1860, ReLJ, BNarm,
FC-1860, ReLU, BMorm, FC-1860,
FC-1860, ReLU, BNorm, FC-186, Zanh/Siegood
ReLU, BNorm, FC-5, Softmax
FC-45, RetU, SNorm, FC-45, Retul, BNorm, FC-45, ReLU, BNorm, FC-45, Tanh/ Bigm//i
FC-22, Retü, BNorm, FC-22, ReLU, BNorm, FC-3, Softmax
Diabetes
7hang, Han, et al. "Self-attention generative adversarial networks." arXiv preprint arXiv:1805.0833 (2018). 1 Miyato, Takenu, al. "Spectral normalization for generative adversarial networks." arXiv preprint arXiv: 180v:1802.05955 (2018).
13

---

Under review as a conference paper at ICLR 2020
C MISSING DATA MECHANISMS
In this paper, we conduct experiments on two mechanisms for missing values: MCAR uniform and MCAR rectangular. As in our experiments and comparisons, we consider the case where only an incomplete dataset is available for training. It is crucial to guarantee that each method has only access to unique incomplete version of each sample. However, it is relatively expensive to load and store feature masks for each sample in the dataset. Instead, we generate missing values during the data load for each batch. A hashing mechanism is used to ensure that the same parts are missing for each sample throughout the training. Note that we set system, python, and external library hash seeds to fixed values to ensure the consisteney between different runs.
Algorithm presents the procedure used for generating missing values with uniform structure. This algorithm is sampling independent Bernoulli distributions with probabilities equal to the missing rate, Algorithm 4 shows the outline for the rectangular missing structure used in image experiments. It consists of selecting a random point as the center of the rectangle and then deciding on parameters to be used for the beta distribution based the missing rate. Finally, the width and height of the rectangular region are sampled from the latent beta distribution. In other words, we generate rectangular regions centered at random locations within the image which have width and height values determined by samples from a latent beta distribution. Here, distribution parameters, a and B, are used to control the average missing rate. The outcome would be rectangular regions of different shape at different locations within the frame with the expected portion of missing area equal to the missing rate.
In order to decide on the beta distribution parameters i.e. and 3 we use numerical simulations. Specifically, we fix one of the parameters and change the other parameter in the range of [1,10], while measuring the average missing rate caused by each case. Figure 5 shows the missing rates caused by different beta distribution parameters. The first half of Figure 5 (missing rates less than about 0.18) corresponds to setting 3 to I and changing values; and the other half fixing to I and changing values. To generate missing rates more than 50% we invert our masks and limit the observation the rectangular region while the rest of the image is missing. Note that missing rates indicate the ratio of features that are missing on the average case. As we are using a latent model for sampling width and height for the rectangles, the actual missing ratios for each specific sample differs between samples. See Table 5 for visual examples of different missing rates and missing structures.
Algorithm 3: MCAR uniform generation.
Input: (complete feature), (missing rate)
Output:
I' (incomplete feature)
seed, t hash
+- Bernoulli shape(r)), = r)
F' < k の (1 k) ④ NaN
Algorithm 4: MCAR rect. generation.
Input: x (complete feature), r (missing rate)
Output: 3m (incomplete feature)
seed, t hash(x)
nz; ny shape(x:)
(px. Py) ~ (uni nz), uni ny))
a, 3 bellaapaaaaa (all_pprrmss()))
params gives a for each missing rate based on numerical
simulations
(w, h) ~ (Beta(a, B) x nz), Beta(a, B) x ny))
+ rect mask(p Py, w.
Fm t- k x + k) NaN
He Kaiming, et al. "Deep residual learning for image recognition." " Proceedings of the IEER conference on computer vision and pattern recognition. 2016. "https: / thub. com www..////uanggg/ygrrrrrrroo
far
14

---

Under review as a conference paper at ICLR 2020
10
A 1, a= [0.10]
1. 10]
8-
6-
alp
4
Simulated
Interpolated
0.2
Missing Rate
Figure 5: Simulation results for measuring average missing rate given different beta distribution parameters.
Table 5: Examples of uniform and rectangular missing structures at different missing rates.
Original
20 %
40 %
60%
80 %
Uniform
Rect.
15

---

Under review as a conference paper at ICLR 2020
D ABLATION STUDY
Figure 6 presents comparison between using (GI W/ Atten.) and not using (GI W/O Atten.) self-attention layers before each residual block in the proposed architecture. We report FID scores on CIFAR-10 with rectangular missingness. As it can be inferred from this comparison, using self-attention achieves consistent improvement over the baseline. We also examined the case of uniform missingness; however, we did observe any significant improvement for this case. One possible explanation could be the fact that imputing missing data with uniform structure can be done by processing local regions and does not require attending to different distant regions across the image.
CIFAR-10 MCAR Rect.
W/ Atten.
30
Gl W/O Atten.
25
FID Score
20
15
10
20 30
70
80
Missing Rate (%)
Figure 6: Comparison of FID scores achieved with (GI W/ Atten.) and without (GI W/O Atten.) self-attention layers CIFAR-10 dataset and rectangular missingness. Lower FID score is better.
Figure 7 shows a comparison of classification accuracies for the Landsat dataset achieved using different ensemble sizes (N), As can be from this figure, higher values of N result in improved accuracies, especially for higher missing rates. Also, it can be observed that for values more than 64 the difference is negligible.
Landsat MCAR Uniform
Accuracy (%)
84
82
Gl (N-128)
Gi (N-64)
Gi (N-16)
Gl (N-4)
25 Missing Rate (%)
Figure 7: Comparison of classification accuracies achieved with different ensemble size (N).
16

---

Under review as a conference paper at ICLR 2020
To study the benefits of the suggested stochastic predictor, conducted experiments comparing Gl with its -on-stochastic variation (N=1). Here, the CIFAR-10 dataset with the rectangular missing structure and missing rates from 20% to as high 90% is used. From Table G can be inferred that as the rate missingness increases, the benefits of the suggested predictor algorithm increase significantly. We hypothesize that higher rates of missingness, the conditional distribution of missing features becomes multimodal. In such scenario, the suggested method captures the uncertainties over the target distribution resulting in the predictor to make more reliable class assignments.
Table 6: Comparison of CIFAR- 10 accuracies for the stochastic (N=128) and the deterministic (N=1) predictor under rectangular missingness.
Accuracy at Missing Rate (%)
Method
20%
40%
60%
70 %
80%
90%
GI (N=128)
84.0
76.9
66.1
59.1
46.0
32.1
GI (N=1)
83.6
75.7
65.1
56.7
42.8
29.4
% difference (normalized)
0.5
1.6
1.5
4.1
6.9
8.4
17

---

Under review as a conference paper at ICLR 2020
E VISUAL COMPARISON
Table panvviiroiide a visual comparison of GI, MisGAN, and GAIN. For each missingness structure, we compare the best two imputation methods based on FID scores in Figure GI versus MisGAN for rectangular missingness and GI versus GAIN for uniform missingness. From Table 7 it can be seen that Gl is more capable in the reconstruction of fine details such as horse legs, car wheels, or plane wings. Regarding the results provided in Table 8, GI imputed samples are generally sharper and more realistic, which is consistent with our hypothesis about the drawbacks of the MSE term in the GAIN objective function.
Table 7: Visual comparison of GI and MisGAN for rectangular missingness. In this visualization, we compare the best two methods for the rectangular missing structure i.c., GI and MisGAN.
Original
Mask
Samples
GI
MisGAN
GI
MisGAN
GI
MisGAN
18

---

Under review as a conference paper at ICLR 2020
Table 8: Visual comparison of Gl and GAIN for uniform missingness. In this visualization, compare the best two methods for the uniform missing structure i.c., GI and GAIN.
Original
Mask
Samples
GI
GAIN
GI
GAIN
GI
GAIN
19

---

Under review as a conference paper at ICLR 2020
F ANALYSIS OF THE RMSE MEASURE
Table presents the comparison of different imputation methods using the RMSE measure on CIFAR-I0 for different missing structures and rates. Generally, RMSE values for the uniform missing structure are lower than their rectangular counterparts. It is consistent with our intuition that imputing uniform missingness is most similar denoising problems where the RMSE measure is frequently used. Additionally, comparing the performance of different imputation methods using the FID measure (Section 4.4) does not demonstrate a clear correlation results shown Table 2 Nonetheless, it is well-known that the FID measure is more suited to measuring the performance of generated images from the underlying distribution (Heusel et al. 2017).
Similarly, in Table IO we provide RMSE values corresponding to experiments on the tabular datasets. Here, GAIN and DAE provide very similar results that are generally better than or MisGAN. This signifies our hypothesis that the MSE loss term may skew generated samples toward the mean of the distribution, resulting in better RMSE values but not necessarily higher final classification accuracies (see Table 2
Table 9: Comparison of imputation RMSE values for CIFAR-10 different missing structures and rates.
RMSE at Missing Rate (%)
MCAR Uniform
MCAR Rect.
Method
20%
40%
60%
20% %
40%
60%
0.026 (::0.003) (033)
0.057 (::.008)
0.090 (=0.006)
0.097 (=0.02)
0.148 (2:0.011)
0.660) (=0.010)
MisGAN
0.079 (d:0.001)
0.161 (d:0.001)
0.257 (2:0.002)
0.106 (#00005)
0.158 (#:0.044)
0.250 (=00001)
GAIN
0.027 (+:0.003) (000003)
0.045 (#:0.001)
0.072 (#0.005)
0.340 (=0.047)
0.511 (#0.001)
0.660 (=0.010)
DAE
0.036 (4-0.001)
0.075 (4:0.002)
0.121 (#:0.005)
0.1 (200007)
0.160 (20.001)
0.233 (=0.029)
Table 10: Comparison of imputation RMSE values for Landsat, MIT-BIH, and Diabetes datasets different missing rates.
RMSE Missing Rate (%)
Dataset
Method
10%
20%
30%
40%
GI
0.040 (#-00005)
0.067 (=0.007)
0.076 (=0.020)
0.136 (t:0.002)
MisGAN
0.068 (+00001)
0.096 (=0.001)
0.118 (e0.001)
0.136 (#:00001)
Landsat Dua & Graff 2017
GAIN
0.018 (10.001)
0.024 (20.001)
0.030 (1:0.001)
0.037 (1:0.001)
DAE
0.020 (#0.001)
0.031 (=0.001)
0.041 (=0.001)
0.052 (#:0000))
0.038 (4:0.001)
0.060 (=0.004)
0.071 (=0.002)
0.095 (1:0.002)
MisGAN
0.073 (#0.007)
0.092 (=0.002)
0.115 (=0.003)
0.111 (#00001)
MIT-BIH Moody Mark
2001
GAIN
0.032 (+0.008)
0.046 (#0.001)
0.055 (#0.004)
0.067 (#:0000))
DAE
0.029 (40.001))
0.048 (=0.008)
0.061 (1:0.009)
0.068 (4:0.00))
0.080 (1.0.022)
0.1 (20. (00.008)
0.149 (1:00000)
0.189 (1:0.09))
MisGAN
0.082 (4:0004))
0.111 (=0.002)
0.133 (=0.001)
0.151 (10.001)
Diabetes Kachuee al. 2019
GAIN
0.064 (#:0001))
0.092 (=0.001)
0.119 (=0.001)
0.140 (#0.001)
DAE
0.065 (4:00001)
0.093 (#0.001)
0.118 (#0.001)
0.143 (#0.001)
20

---

Under review as a conference paper at ICLR 2020
G IMPACT OF TRAINING NOISE
Addition of noise to input vectors often serves as an input augmentation and results improved generalization accuracies. In order verify that the improved GI performance to the introduction of noise in the suggested architecture, we conducted an experiment by adding different amounts of Gaussian noise during the training process for GAIN and GI. Specifically, compared how the CIFAR-10 test accuracies change at different degrees of training noise for uniform and rectangular missingess structures the average missing rate of 40%.
merely due
According to Table 11 adding small amounts of Gaussian noise (e.g., std=0.0125) improves the generalization under uniform missingness for both GI and GAIN. Even this case, GI is still outperforming GAIN in terms of final classification performance. It is also interesting to point out that for the case of rectangular missingness adding Gaussian noise results a consistent reduction in the classification accuracy for both methods.
Table 11: Top-1 CIFAR-10 classification accuracy at 40% missing rate using added training noise.
Accuracy (%)
MCAR Uniform (40 %)
MCAR Rect. (40%)
Noise STD
GI
GAIN
GI
GAIN
0.0
87.1
86.0
76.9
73.6
0.0125
87.3
86.3
76.8
73.3
0.025
86.5
86.6
76.7
73.2
0.05
85.6
84.7
73.7
72.4
0.1
82.0
80.6
68.7
67.0
21

---

Under review as a conference paper at ICLR 2020
H IMPACT OF THE MSE LOSS TERM
In our earlier discussions, we stated that the MSE loss term used in GAIN would bias the distribution of generated samples toward the mean of the distribution. Here, synthesized dataset is used to illustrate the impact of MSE loss term on the distribution of generated samples, A hyperparameter, X, controls the weight of the MSE term in the final objective function. As can be observed from Figure 8, the higher the À parameter, the lower the variance of the generated samples (i.e., more bias toward the mean of the distribution).
Original (o=0.13)
0.14
0.30
Freq.
0.08
0.05
0.04
0.00
0.0
0.2
0.4
0.6
0.8
1.0
À = 0 (0=0.17)
À = = (0=0.09)
0.2
Freq.
0.1
Freq.
0.1
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
X
X
A 10 (o=0.08)
À = 10+1 (o=0.03)
0.4
0.4
Freq.
Freq.
0.2
0.2
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
X
X
(b)
Comparison of generating samples from Gaussian distribution (a) samples from the original distribution, (b) samples generated using GAIN imputers with different significance of the MSE term (controlled by A).
Figure 8:
22

---

Under review as a conference paper at ICLR 2020
1 IMPACT OF THE DISCRIMINATOR HINT VECTOR
Yoon et al. 2018) suggested the idea of guiding the discriminator network using a hint mechanism. hint vector reveals subset of features that are missing to the discriminator. Figure and I0 provide comparison of learning curves for GI implemented using different hint rates. From Figure 2 using the hint mechanism does not result any noticeable improvement in the final imputation quality justifying the added complexity. For the case of the rectangular missing structure in Figure TO] however, using the hint vector causes instabilities the training process. One possible explanation is: providing even small portion of the mask hint, due to the deterministic nature of the rectangular shape equivalent providing region boundaries the discriminator making it obvious for the discriminator. In GAN training generally want to have equal competition between the generator and discriminator.
CIFAR-10 10 MCAR Uniform
hint rate 0.0
hint rate 0.3
hint rate 0.5
200
FID Score
100
0
25
50
75
100
125
150
175
Epoches
Figure 9: Learning curves for CIFAR- 10 with uniform missing structure at different discriminator hint rates.
CIFAR-10 MCAR Rect.
hint rate
hint rate 0.05
hint rate 0.1
100
FID Score
80
60
20
50
125
150
175
Epoches
Figure 10: Learning curves for CIFAR-10 with rectangular missing structure at different discriminator hint rates.
23


Reviews:
[{"review_id": "Hyxvxaag5H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to impute missing features using a generative model and train a predictive model on top of imputed dataset to improve classification results. They first train a GAN model where the generator outputs an imputed representation of the input and discriminator is trained to predict if an individual features (such as a pixel) is imputed or not. Given the generator and incomplete sample, they train a predictor using the output of the generator, imputed sample, as input. Their main contribution is using a MC averaging to compute the prediction by repetitively sampling from the noise variable, z, and generating different imputations from generator. They show that the proposed model improves upon the previous SOTA on final classification performance.\n\nOverall the paper is clearly written. But I do feel it is a bit incremental over the GAIN approach. The overall GAN architecture is very similar to GAIN's and although stochastic prediction shows clear improvements it is a bit straightforward. However, I think the uncertainty of the imputations and its effect on the final prediction is interesting. I suggest the authors to extend this part with more detailed analysis.\n\nThere are several parts that are confusing/missing in the paper:\n\n- In GAIN, they use a hint vector as an input to the discriminator. They show that without the hint vector, there is no unique solution (this is shown without the MSE loss). The authors do not use this vector in their approach (as in Figure 1) and it is not clear to me if it causes any instabilities or if multiple experiments yield similar results or if the stochastic prediction benefits from this.\n- On what type of examples GI is more accurate than other models? Since stochastic prediction is the main difference from GAIN, is this related to the multi-modality of the noisy examples?\n- Can you explain the difference between the results in Figure-7 and Table-2? Results between the two mismatch.\n- I think the statement in the first paragraph in Section 4.4 that \"MSE loss term would act as a denoising loss smoothing noisy missing pixels\" could be misleading. MSE is used with mask in GAIN, hence it only applies to the observed features during training. Its effect on smoothing noisy missing pixels is not clear.\n\n\nI think the paper would benefit if the authors could explain/show:\n- Increasing the missing rate would also increase the possibility that the ground truth be a more multi-modal distribution. Especially in rectangular generation part where it can remove a complete object. Does stochastic averaging benefit more in this case?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HyeAuZ3DtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The uncertainty of having a missing value is investigated on the prediction by not assigning a single imputed value but N different values generated via an imputer network (based on GAIN). Unlike old-fashion multiple imputation techniques, one predictor is trained on different samples and it induces the uncertainty. The experiments on number of datasets show the proposed predictor is capable of having a fairly better performance.\nOverall, this paper raises an interesting point about missing data imputation via generative models, and well-written; however, there are number of concerns:\n1-\tThe predictor is trained on different version of imputed samples (imputed via the generator); this equates to making noisy version of the real samples, where noise is applied to the missing variables. A side effect of this is generalization of the predictor; thus, have you been careful that the improved accuracy is not due to generalization? In other words, if we imposed the generalization via adding gaussian noise to the imputed samples by GAIN for example, would we get improved accuracy too?\n2-\tYour method known as GI is a modified version of GAIN. You could also use MisGAN, and I am wondering if the results would have been different if generator in MisGAN was used in GI in Figure 2 (b), as MisGAN works better than GAIN.\n3-\tI am also wondering how the GAIN imputation changes by removing the MSE term? GI discards the MSE term in GAIN, and it changes the distribution of the imputed variables by GAIN. Could you maybe fit a Normal distribution on a chosen imputed variable  (N=128) and visualize how different it is from the distribution of imputed variables with GAIN with MSE term. \n4-\tIn justification for claim 1, it is said “This is equivalent to training models using noisy labels”. This is not accurate: in noisy label prediction, we have one (noisy) y corresponding to each x, in your case there are multiple ys for one sample.\n5-\tIn the implementation details, I cannot fully wrap my head around the part “z vector of size 1/8”; how did you choose this 1/8?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "B1xLXZ319r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is a nice piece of incremental work on top of previously published GAN imputation methods. It seems to work well in the limited evaluation and is at least claimed to be easier to use for practitioners. This paper could benefit tremendously from both better evaluation and discussion. The paper would be much clearer if GI contextualized itself relative to GAIN on the one hand (which is the most similar GAN method) and multiple imputation on the other hand (of which this is almost, but not quite, an instance of). \n\n\nSuggestions for improving the introduction & discussion: \n* The purpose of this paper is to model uncertainties about missing values — you really should say more about probabilistic methods than \"A few exceptions exist such as Bayesian models”.  At least give some motivation for why certain imputations problems couldn’t be feasibly solved by modeling the missing values in a probabilistic programming framework. \n* Other GAN methods for imputation (GAIN and MisGAN) are dismissed as \"often very complicated to be applied in practical setups by practitioners”. Given that the described method resembles GAIN, is it really much simpler? If so, can you be more specific when characterizing related work?\n* \"This is different from approaches such as multiple imputation where several predictors are trained on different imputed versions of a dataset.” — the main difference between this approach and MI is that you’re interleaving imputation and training a downstream model. Emphasize this earlier on, since it will make the whole technique easier to understand.\n\nSuggestions for improving the evaluation:\n* You’re imputing missing rectangles from an image dataset — please show us the resulting images. I would greatly prefer this to Section 4.5 — which is a very low sample size, low dimensionality example and it’s really unclear how well it generalizes to real data. \n* \"We also considered using root means squared error (RMSE); however, we decided not to use this measure as we observed an inconsistent behavior using RMSE in our comparisons as RMSE favors methods that show less variance rather than realistic and sharp samples from the distribution.” — I think this is a mistake, likely motivated by the proposed method doing worse under the RMSE metric. Show us several relevant metrics and then discuss their tradeoffs afterward. \n* \"We run each experiment multiple times (at least 4)” — please report how often each experiment was run, even better if you standardize this number. \n* For Table 2, please provide accuracy without missing values as a baseline.\n* Add MICE or some other “standard” imputation method as a baseline. \n\nSuggestions for improving readability: \n* Many sentences start with “in this” (e.g. “in this case”, “in this setting”, &c). Sometimes these sentences even co-occur within the same paragraph. Try to switch up the phrasing and move away from repetition. \n* Not a complete sentence: \"For instance, jointly training multiple generator/discriminator networks, tuning objective functions with multiple hyper-parameters, etc.\"\n\nUpdate: I think the latest draft of the paper is a big improvement, the inclusion of a \"classical\" baseline, improved language and additional appendices are all welcome. I'm leaving the rating as a \"weak accept\" since the paper still feels rough and could use additional editing/streamlining. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method that does uncertainty modeling over missing data imputation using a framework based on generative adversarial network. While the method shows some empirical improvements over the baselines, reviewers have found the work incremental in terms of technical novelty over the existing GAIN approach which renders it slightly below the acceptance threshold for the main conference, particularly in case of space constraints in the program. 
Decision: Reject