EMFlow: Data Imputation in Latent Space via EM and Deep Flow Models
Authors: Qi Ma
Sujit K Ghosh
Venue: ICLR 2022 Submitted (2022)

Abstract: The presence of missing values within high-dimensional data is an ubiquitous problem for many applied sciences. A serious limitation of many available data mining and machine learning methods is their inability to handle partially missing values and so an integrated approach that combines imputation and model estimation is vital for down-stream analysis. A computationally fast algorithm, called EMFlow, is introduced that performs imputation in a latent space via an online version of Expectation-Maximization (EM) algorithm by using a normalizing flow (NF) model which maps the data space to a latent space. The proposed EMFlow algorithm is iterative, involving updating the parameters of online EM and NF alternatively. Extensive experimental results for high-dimensional multivariate and image datasets are presented to illustrate the superior performance of the EMFlow compared to a couple of recently available methods in terms of both predictive accuracy and speed of algorithmic convergence.

Under review as a conference paper at ICLR 2022
EMFLOW: DATA IMPUTATION IN LATENT SPACE VIA EM AND DEEP FLOW MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
The presence of missing values within high- dithesiieniii data is an ubiquitous problem for many applied sciences. serious limitation of many available data mining and machine learning methods is their inability to handle partially miss- ing values and so an integrated approach that combines imputation and model estimation is vital for down-stream analysis. A computationally fast algorithm, called EMFlow, is introduced that performs imputation in a latent space via an online version of izing flow (NF) model which maps the data space to a latent space. The proposed EMFlow algorithm is iterative, involving updating the parameters of online EM and NF alternatively. Extensive experimental results for high-dimensional multi- variate and image datasets are presented illustrate the superior performance of the EMFlow compared to couple of recently available methods in terms of both predictive accuracy and speed of algorithmic convergence.
(EM) algorithm by using a normal-
INTRODUCTION
Missing values often encountered for real-world datasets and are known to adversely impact the validity of down-stream analysis. Most machine learning (ML) algorithms and statistical tools often ignore the problem of partially observed features by simply dropping entire cases with miss- ing values, which could result into biased estimation and underestimation of parameter uncertainty (Schmitt al., 2015; Somasundaram & Nedunchezhian 2011: Lall 2016).
There have been some recent attempts to develop ML methods that properly handle missing values. Early works this field performed imputation in a supervised manner where a complete training set is needed to learning the correlation between missing and observed entries (e.g. LarnciaL jet al., 2010, Rezende et al. 2014 Bertalmio et al.) (2000; Xie et al., 2012, Yeh al. 2016). How- ever, is common that large collection of fully observed data is hard to acquire, which makes those data-greedy supervised algorithms less effective. On the other hand, early attempts in unsupervised imputation methods, such as the collaborative filtering (Sarwar et al.) 2001), usually learn the corre- lation across dimensions by embedding the data into a lower dimensional latent space linearly (e.g. Little & Rubin 2019 Audigier et al.) 2016). To improve the limited representation power of those linear models, kernel- based methods are also proposed but at the cost excessive computational
load when dealing with large datasets (e.g. (anguinetti & Lawrence 2006 Liu al. 2016).
More recently, several imputation methods based on deep generative models have been proposed, providing much more accurate estimates of missing values especially for high-dimensional image datasets. In this work, we introduce EMFlow that integrates the normalizing flow (NF) (Dinh et al.)
2016 Rezende & Mohamed 2015 Dinh al., 2014; Kingma & Dhariwal 2018) with an online version of Expectation- Mixitationn-Mxmizzzttiii (EM) algorithm (Cappé & Moulines 2009). The proposed framework motivated by the strength and weakness of EM. a class of iterative algorithm de- signed for latent variable models, EM can be applied to data imputation in an interpretable way (Little & Rubin 2019). Additionally, the learning of EM usually numerically stable and its con- vergence property has been studied extensively (e.g. Ma et al. 2000 Zhao et al. 20200] Meng al.) 1994 Wu 1983). However, the E-step and M-step are only traceable with simple underlying dis- tributions includie hucluding multivariate Gaussian or Student's distributions well their mixtures (e.g.
Di Zio et al. 2007 xian Wang et al., 2004). On the other hand. NF is capable of latent representa- tion and efficient data sampling, which makes it convenient bridge between the data space and the

---

Under review as a conference paper at ICLR 2022
latent space. Therefore, we let EM perform imputation in the latent space where simple interrfeature dependency is assumed (i.e. the underlying distribution multivariate Gaussian). Meanwhile, NF is used to recover the mapping between the complex inter-feature dependency in the data space and the one in the latent space learned by FMM
The inference of EMFlow adopts iterative learning strategy that has been widely used in model- based multiple imputation methods where the initial naive imputation is refined step by step until convergence (e.g. Gondara & Wang 2017 Buuren & Proothuis-Oudshooee Bühlmann 2012). Specifically, three steps are performed alternatively:
2010 Stekhoven &
update the density estimation of complete data including the observed and current imputed
values;
update the base distribution (i.c. f and Σ) in the latent space by EM; and
update the imputation in the data space.
Note that the first update corresponds to optimizing the complete data likelihood as well as the reconstruction error computed on the observed entries. We also derive an online version of EM such that it only consumes a batch of data at time for parameter updates and thus can work with deep generative models smoothly. We will show that such learning schema has simpler implementation and leads to faster convergence compared to other competing methods.
The main contributions of this work are
an imputation framework combining an online version of ExpectationMMxxiiiaatiee Maximizzaiinn (EM) algorithm and the normalizing flow;
(ii)
iterative learning schema that alternatively updates the inter-feature dependency in the latent space (i.e. the base distribution) and density estimation in the data space;
(ii a derivation of online EM in the context of missing data imputation; and
(iv) extensive experiments on multiple image and tabular datasets to demonstrate the imputation quality and convergence speed of the proposed framework.
RELATED WORK
Recently, the applications of deep generative models like Generative Adversarial Networks (GAN) [Goodfellow et al. 2014) have been extended to the field of missing data imputation under the as- sumption of Missing at Random (MAR) or Missing Completely at Random (MCAR). GAIN (Yoon) et al. 2) designs a compete data generator that performs imputation, and discriminator to dif- ferentiate imputed and observed components with the help hint mechanism. MisGAN (Li et al. 2019) introduces another pair of generator-discriminator that aims to learn the distribution of miss- ing mask. However, training GAN- based models is a notoriously challenging for its excessively complex structure and non-convex objectives. For example, MisGAN optimizes three objectives jointly involving six neural networks for imputation. Furthermore, GAN-based models do not ob- tain explicit density estimation that could be critical for down-stream analysis (e.g. (Ferdosi al.
2011 Zambom & Ronaldo 2013).
Some imputation techniques based on Variational Autoencoders (VAE) KKngmma & Welling 2013)
are also developed. For example. MIWAE Mattei & Frellsen (2019) leverages the importance- weighted autoencoder (Burda et al. 2015)) and optimizes a lower bound of the likelihood of observed data. But the zero imputation adopted by it can be problematic since observed entries can also be zero. It also needs quite large computational power make the bound be tight. EDDI (Ma) et al., 2018) avoids zero imputation by introducing a permutation invariant encoder. Imputation under Missing not at Random (MNAR) has also been explored by explicitly modeling the missing mechanism (Ipsen al. 20200] Collier et al. (220).. However, all VAE-based approaches only permit approximate density estimation no matter how expressive the inference model is.
Our proposed framework builds on the work of MCFlow (Richardson et al. 2020 that utilizes NE to learn exact density estimation on incomplete data via an iterative learning schema. The core component of MCFlow is feed forward network that operates in the latent space and attempts to find the likeliest embedding vector in a supervised manner. Although MCFlow achieves impressive performance compared to other state-of-art methods, it remains ambiguous that how the feed forward
2

---

Under review as a conference paper at ICLR 2022
network exploits the correlation in the latent space. A key difference between the MCFlow and our proposed method is that we exploit the correlation in the latent space which in turn induces correlation in ambient space via NF. We also find that the inference of MCFlow often has slow convergence speed and can be unstable.
3 APPROACH
3.1 PROBLEM DEFINITION
We define the complete dataset X a collection of p-dimensional vectors {x Xn) that are independent and identically distributed (i.id.) samples drawn from px in a p-dimensional data space X. The missing pattern of x; is described by a binary mask m; 1), such that Xij is missing if m,j 1, and Xij is observed if m,j = 0.
Let x" and x be the observed and missing parts of x, and Pwr (m/x) PM (m[[x xx") be the the missing
conditional distribution of the mask. Based on dependency between m and mechanism can be classified into three classes (Little Rubin 2019):
MCAR: Pas (m x", ,x*) Pas (m)
MAR: Pa (m[x², xx") = PM (m|x*)
MNAR: The probability of missing depends on both x" and xm
Throughout this paper, we only focus on MCAR or MAR where the missing mechanism can be safely ignored. The typical objective learning a latent model is maximize the observed data log-likelihood defined as
Σ log /
L abs (0)
px (x,, x,"; 0) dx.".
However, our goal in this work is to estimate from incomplete data as well as obtain accurate imputation under px. Therefore, we attempt to learn the reconstructed data X (Ri.... 8,,² and the estimated model parameter via
22
(x
arg max x.EX
log px (x100
where X S X is the search space for x; where the observed locations have fixed values.
3.2 NORMALIZING FLOWS
To make it possible to optimize equation P.x (x; 0) needs to be specified in parametric way that should be expressive enough, as the density is potentially complex and high-dimensional. To this end, NF to model px (x; 0) invertible transformation fo of base distribution pz (z; Φ) in the latent space 2. Under the change of variables theorem, the complete data log-density is specified as
@5 (x)
(x = P2(f (x): det
(3)
Ox
where = (v. Φ).
fø is usually composed by sequence of relatively simple transformations to approximate arbi- trarily complex distributions with high representation power. In this work, we choose Real NVP Dinh et al. 2016) based affine coupling transformations as the flow model Recently, Teshimal
See appendix A for the details of Real NVP.

---

Under review as a conference paper at ICLR 2022
al. 2020 has shown that flow models constructed from affine coupling layers can be universal distributional upproximators.
Data imputation relies the inter- dependency that is usually intractable and hard to capture the data space X' with the presence of missing entries. Therefore, we make the following two assumptions related to NF.
The inter-feature dependency in the latent space Z is simple and can be character- ized by multivariate Gaussian density, that is:
Assumption 1:
pz Φ) N p. 2)
where Φ = (1,, Σ) consists latent parameters; mean vector μ and covariance matrix Σ.
Note that the base distribution is usually chosen as a standard Gaussian distribution (with 0 and E Ip) the literature for simplicity. However, the covariance Σ essential in our imputation work to represent the inter-feature dependency.
Assumption 2: Given that the transformations involved in NF are feature-wise (e.g. Dinh et al. 2016 2014), expect NF to learn the mapping between the complex inter-feature dependency in the data space X' and the simple one in the latent space Z.
3.3 ONLINE EM
EM is class of iterative algorithms for latent variable models including missing data imputa- Little & Rubin (019). In EMFlow, it works in the latent space Z where the underlying dis- tribution (2; u4, 2). Given the embedding vectors (2i,...,20 .2.} where z, (x:), and the corresponding missing mask (mı,...; m,), EM aims to estimate (11, Σ) an iterative way:
((++1) = 9 ( 500; {z,, m
(5)
S((11) =p 500;; [z, m. H
where (f((0, E(0) are the estimates at the 1th iteration, and (x)())))))) denote the mappings between two consecutive iterations?
Given estimates (ūz Σ). the missing part z!" is imputed by its conditional mean given the observed part z
Pm, + E Em,o, (So.o) )
-1 (2f
4
E
Σ)
iro))
(6)
where is the observed mask (i.e. the complement of m,), and the subscripts of (Øz, Σ) denote the slicing indexes.
When processing datasets of large volume, EM becomes impractical because it needs read the whole data into the memory for each iteration. Following the framework introduced by Cappé & Moulines 2209)), we derive an online version of EM algorithm in the context of data imputation. Let B C [c,...,,n denote mini-batch of sample indexes, the online EM first obtains local estimates from batch of data:
Poocat 9b (T)))) S(1); (zi, m:lien) Dtoeat = gx (1) 500; (z;, m:hee)
(7)
The global estimates are then updated in the fashion of weighted average:
(1-1) = -P+11 + (1 Pt+1 (1)
(8)
(1-1) Pt+1 Etocal + Pt+1] {(1)
"ee appendix B for the details of Iu and gr.
4

---

Under review as a conference paper at ICLR 2022
X in
Initial Imputation
Data Space !'
X
NF f (
NF fo(-)
Z
Online EM
Latent Space Z
Figure EMFlow architecture.
where (P1,P2,...) are sequence of step sizes satisfying
0 < < Σ
P: 0o and
2 P
<
x
(9)
1=1
1=1
In this work, we use a step size schedule defined by
Ct t= 1,2,...
(10)
where a positive constant and y E (0.5, 1].
3.4 ARCHITECTURE AND INFERENCE
EMFlow is composite framework that combines NF and online EM. As illustrated in Fig I NF is the bidirectional tunnel between the data space and the latent space, aiming to learn the complete data density px. In the latent space, the online EM estimates the inter-feature dependency of the embedding vectors and performs imputation. To address the issue that NF needs complete data vectors for computation, the incomplete data X are imputed naively (e.g. median imputation for tabular datasets) at the very beginning to get the initial current imputed data X. Afterwards, the objective in equation is optimized in an iterative schema, where each iteration consists of training phase and re-imputation phase.
Training Phase At this phase, the current imputed data X stay fixed, while the parameter estimates of NF (i.e. b and base distribution (i.e. je and Σ) are updated in different ways.
First of all, given the current estimated base distribution (N; 2, E), the flow model fe are learned by minimizing the negative log-likelihood of batch of the current imputed data X
(i =
Σ (R; IEB
(11)
where [B] denotes the batch size.
The computation of L requires exact likelihood evaluation that is equipped by NF. Once the flow model parameters get updated, we obtain the embedding vectors in the latent space:
2i =
-=
EE B
(12)
5

---

Under review as a conference paper at ICLR 2022
Although the embedding vectors are complete, they treated as incomplete using the missing masks in the data space [m; heB. given that the invertible mapping parameterized by fg is feature- wise. Therefore, the online EM imputes the missing parts of the embedding vectors with the current global estimates (it, E):
2 B(z, pe; D), i
(13)
which results in new embedding vectors {E) where 2; consists of the observed part z and the imputed part "," After the imputation, the global estimates (j,, Σ) are also updated following equation 7 and equation
Since the base distribution has been changed, it's necessary to update the flow model fy again by optimizing composite loss:
L2(() = [3|
logpx(X;; P) - al (x, m,)
(14)
EE
where x, = (2,)) and Lico (X,, & m.) is the reconstruction error only for non-missing values:
P
[xe(X, X,, m,) ) = > m;,)(xi, R,y))
(15)
jml
In this composite loss, the first term forces the reconstructed data vectors {x} to have high likelihood the data space, while the second term encourages {e to match the observed parts of the incomplete training data. And since {Rhe are transformed from {ãheB via fø. both terms are conditioned on the inter-feature dependency learned by EM. A pseudocode for this training phase is presented in Algorithm [ and the implementation details that aim to make the training phase more stable is presented in appendix D
Algorithm Training Phase
1: Input: Current imputation: X (R....., 8², missing masks M (m....., m,," initial estimates of the base distribution (7) (01), online EM step size sequence: Pl P2. Pt.
2: for 1 to Tepoch do
3: Get a mini-batch X = (8, JieB
4: update the flow model Compute L2 in equation I1 Update z via gradient descent
5:
6:
7:
# update the base distribution
8:
2, T (R.); i B
Impute in the latent space with (7(t-1), S(e-1) to get {2, JeB via equation 13
9:
10:
Obtain updated (7)(), ((0) via equation and equation update the flow model again
12: X; Fe (2:), E
Compute L2 via equation 14
14: Update dà via gradient descent
Re-Imputation Phase After the training phase, the re-imputation phase executed to update the current imputation X. This procedure is similar to that in the training phase, except that all model parameters are kept fixed. As shown by the last line of Algorithm 2 the missing parts of X are replaced with those of the reconstructed data vectors, while the observed parts are kept the same.

---

Under review as a conference paper at ICLR 2022
Algorithm 2 Re-imputation Phase
1: Input: Current imputation: X (K1,...p Xn )², missing masks M (m (mrr mn) ) esti-
mates the base distribution (ji, 2) from the the previous training phase
2: for i = to do
3:
2, = fw (xx)
Impute in the latent space with (ūz, Σ) to get 2; via equation 13
4:
5:
X; (2.)
6:
update current imputation X, @ m,
7:
X
Table 1: Imputation results on UCI datasets RMSE (lower is better).
MCAR
MAR
Data
EMFlow
MCFlow
GAIN
EMFlow
MCFlow
GAIN
News
.139 + .001
.16 + .0015
.197 + .005
.172 - .000
.181 = .001
.211 + .039
Air
.097 .005
.11 .004
.17 .005
.040
.001
.055
.001
.061
.023
Letter
.111 + .001
.121 + .00
.127 +001
.110 1001
.127 + .001
.166 + .040
Concrete
1447 + .004
.233 + .007
.14 .008
.133
.006
.198 + .010
.184 + .012
Review
.229 0003
.234 - .004
.278 .005
.194 = .004
.99 + .004
.234 + .014
Credit
.125 .001
.135 + .002
.131 .002
.024 + .001
.028 1 .002
.029 + .002
Energy
.086
.001
.092 - .001
.110 .002
.175 1 .002
.176 .002
.250 .019
CTG
.104 .006
.006
.14 + .005
.143 1 .008
.105 0001
.153
.003
.165 .006
Song
.025 + .000
.030 + .006
.034 = .002
.024 = 0000
.000
.031 + .014
.028 + .001
Wine
.076
.001
.098 + .002
.097 + .002
.02 - .002
.124 .003
.135 .007
Web
.001 .000
.002 = .000
.003 .003
.002 .004
.006 .009
.006 .010
Online News Popularity
Air Quality
Appliance Energy
60
50
Training Loss
25
1.0
20
-25
0
-50
-10
-75
-40
0.250
0.6
MCFlow
0.5
EMFlow
0.3
RMSE
0.200
0.175
0.3
0.2
0.2
0.1
250
500
1000
100
200
400
200
400
600
Elapsed Time /
Elapsed Time /
Elapsed Time i
Figure 2: Comparison of convergence speed in terms of the training loss and test set RMSE on three UCI datasets.
4 EXPERIMENTS
In this section, we evaluate the performance of EMFlow on multivariate and image datasets in terms of the imputation quality and and the speed of model training. Its performance is compared to that of MCFlow, the most related competitor that has been shown to be superior to other state art methods (Richardson et al. 2020).

---

Under review as a conference paper at ICLR 2022
Table 2: Imputation results image datasets RMSE (lower is better)
Missing Rate
.1
.2
.6
.7
.8
GAIN
1029
.1844
.1399
.1455
.723
.1744
.1677
.2000
.210
MisGAN
.0833
.1177
.1844
.1227
.1311
.1888
.1512
.1906
.2621
MNIST
MCFlow
.0835
.0879
.0894
.0941
.1027
.1199
.1251
.1463
.2000
EMFlow
.0726
.0775
.0832
.0901
.0986
.1100
.1600
.1504
.1951
GAIN
.1225
.1090
.1103
.1073
.1094
.2022
.1177
.1266
.5388
MisGAN
.1577
.134
.1478
.1326
.1588
1824
2036
.2660
.3011
CIFAR-10
MCFlow
.1083
.1112
.1179
.1273
.1340
.1387
.1466
.1552
.1702
EMFlow
.0444
.0479
.0525
.0575
.0619
.0689
.0782
.0926
.1888
Table 3: Classification accuracy imputed image datasets (higher is better)
Missing Rate
.1
.2
.3
.6
.7
.8
MCFlow
.9944
.9888
.9788
.971
.9400
.9866
.9659
.9311
.7732
MNIST
EMFlow
.9894
.9844
.9822
.9878
.9860
.9244
.9696
.9253
.7502
MCFlow
.8322
.7081
.5225
.4666
3406
.2820
.2476
.2194
.1875
CIFAR-10
EMFlow
.9855
.8944
.8883
.8355
.8166
.7446
.6214
.4868
.3127
To make the comparison more objective, both models use the same normalizing flow with six affine coupling layers. We also follow the authors' suggestion for the the hyperparameter selection of MCFlow throughout this section. Additionally, also present the benchmarks of other state-of-art models including GAIN Yoon et al. 2018) and MisGAN (Li et al. 2019) for more comprehensive comparisons.
4.1 MULTIVARIATE DATASETS
Ten multivariate datasets from the UCI repository (Dheeru & Taniskidou 2017) used for evalua- tion. For all of them, each feature is scaled to fit inside the interval [0, 1] via min-max normalization. We simulate MCAR with a missing rate of 0.2 by removing each value independently according to a Bernoulli distribution. We also simulate MAR scenario where the missing probability of the last 30% features depends on the values of the first 70% features
The initial imputation is performed by randomly sampling from the observed entries of each feature. All experiments are conducted using five-fold cross validation where the only goes through the re-in re-imputation phase in each iteration. The choices of hyperparameters are detailed in appendix where we also show that EMFlow not sensitive the choice of hyperparameters.
Results The imputation performace is evaluated by calculating the Root Mean Squared Error (RMSE) between the imputed and true values. As shown in Table [ EMFlow performs constantly better than MCFlow under both MCAR and MAR settings for nearly all datasets.
Additionally, We trained EMFlow and MCFlow on the same machine with the same learning rate and batch size to compare the convergence speed. Figure (quure[]shar shows the training loss and the test set RMSE over time on three UCI datasets. It shows that EMFlow converges significantly faster than MCFlow. In fact, EMFlow converges within three iterations for most of the UCI datasets.
4.2
IMAGE DATASETS
We also evaluate EMFlow on MNIST and CIFAR-10. MNIST is a dataset of 28x28 grayscale images of handwritten digits (LeCun al. 1998), and CIFAR-IO a dataset 32x32 colorful images from 10 classes (Krizhevsky et al. 2009). For both datasets, the pixel values of each image are scaled to
'eee appendix P for the details of how MAR is simulated.
8

---

Under review as a conference paper at ICLR 2022
Missing Rate = 0.5
Missing Rate 0,9
Ground True
Observed
MCFlow
EMFlow
Figure 3: Sample imputed images for CIFAR-IO at missing rates of 0.5 and 0.9.
[0, 1]. In this section, we simulate MCAR where each pixel is independently missing with various probabilities from 0.1 0.9.
The initial imputation is performed by nearest-neighbor sampling where a missing pixel filled by one of its nearest observed neighbors. In our experiments, the standard 60.000/10,000 and 50,000/10,000 training-test set partitions are used for MNIST and CIFAR-10 respectively. The choices of hyperparameters are detailed in appendix
Results shows the RMSE of all considered methods on both image datasets. In the case of MNIST, EMFlow and MCFlow have similar RMSE and outperform other methods, while MCFlow starts to gain slight advantage under high missing rates. In the case of CIFAR-10, EMFlow achieve much lower RMSE than all competing methods.
To further demonstrate the efficiency of EMFlow, We also compare EMFlow and MCFlow with respect to the accuracy of post-imputation classification. For this purpose, a LeNet-based model and VGG19 model were trained on the original training sets of MNIST and CIFAR-10 respectively. These models then made predictions on the imputed test sets under different missing rates. Table 3 shows that EMFlow yields slightly better post-imputation prediction accuracy than MCFlow MNIST, while the improvement is much more significant on CIFAR-10. We note that these findings in good agreement with the RMSE results.
To qualitatively compare the imputation quality of EMFlow and MCFlow, Figure 3 shows sample imputed images from CIFAR-10 with MCAR missing rates 0.5 and 0.9. The first row includes the (complete) ground truth images for reference, while the second row includes the (incomplete) observed images on which the models were trained. The last two rows showcase the reconstructed images by MCFlow and EMFlow, respectively. It's clear that EMFlow performs better than MCFlow by recovering more details and displaying sharper boundaries and cleaner background.
CONCLUSION
We propose novel architecture EMFlow for missing data imputation. combines the strength of the online EM and the normalizing flow to learn the density estimation in the presence of incom- plete data while performing imputation. Various experiments with multivariate and image datasets show that EMFlow significantly outperforms its state-of-art competitor with respect to imputation accuracy as well the convergence speed under a wide range of missing rates and different missing mechanisms. The accuracy of post-imputation classification on image datasets also demonstrates superior EMFlow's ability of recovering semantic structure from incomplete data.
REPRODUCIBILITY STATEMENT
The code and data reproduce the results in this work included in the supplementary materials.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Vincent Audigier, François Husson, and Julie Josse. Multiple imputation for continuous variables using a bayesian principal component analysis. Journal of statistical computation and simulation, 1111::2224211550 2016.
Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image inpainting- In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pp. 417 2000.
AE Brockwell. Universal residuals: A multivariate transformation. Statistics & probability letters, 77(14):14331477, 2007.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint arXiv:1509.000 htt::::090000110 00599, 2015.
Stephen Burgess, Ian R White, Matthicu Resche-Rigon, and Angela M Wood. Combining multiple imputation and meta-analysis with individual participant data. Statistics in medicine, 32(26): 4499-4514, 2013.
S van Buuren and Karin Groothuis- Oudshoon. mice: Multivariate imputation by chained equations in r. Journal of statistical software, pp. 1-68, 2010.
Olivier Cappé and Eric Moulines. On-line expectation expecatton-maxxiizzatoo algorithm for latent data mod- els. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 71(3):593-613 2009.
Yilun Chen, Ami Wiesel, and Alfred O Hero. Robust shrinkage estimation of high-dimensional covariance matrices. IEEE Transactions on Signal Processing, 59(9):497744107, 2011.
Mark Collier, Alfredo Nazabal, and Christopher KI Williams. Vaes in the presence of missing data. arXiv preprint arXiv:2006.0530), 2020.
Dua Dheeru and Karra Taniskidou. Uci machine learning repository. 2017.
Marco Di Zio, Ugo Guarnera, and Orietta Luzi. Imputation through finite gaussian mixture models. Computational Statistics Data Analysis, (111))5305--336, 2007.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. arXiv preprint arXiv: 1XXXx::1100.800 1410.85/6 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv: 08803, 2016.
Jianqing Fan, Yingying Fan, and Jinchi Lv. High dimensional covariance matrix estimation using factor model. Journal of Econometrics, 147(1):188) 1877 2008.
BJ Ferdosi, H Buddelmeijer, SC Trager, MHF Wilkinson, and JBTM Roerdink. Comparison of density estimation methods for astronomical datasets. Astronomy & Astrophysics, 531:A114, 2011.
Pedro Garcfa-Laencina, José-Luis Sancho-Gómez, and Aníbal R Figueiras-Vidal Pattern classifi- cation with missing data: review. Neural Computing and Applications, 19(2):263-222, 2010.
Lovedeep Gondara and Ke Wang. Multiple imputation using deep denoising autoencoders. arXiv
preprint arXiv:/705.n0 700.02237 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair,
Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Niels Bruum Ipsen, Pierre- -eere-Alexandre Mattei, and Jes Frellsen. not-miwae: Deep generative mod- elling with missing not at random data. arXiv preprint arXiv:2006. /287/, 2020.
10

---

Under review as a conference paper at ICLR 2022
Diederik P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. arXiv preprint arXiv: ar7.v:1807.030339, 2018.
Diederik Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding variational bayes.
arXiv preprint
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ranjit Lall. How multiple imputation makes a difference. Political Analysis, 24(4):414-433, 2016.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11)22278-3224 1998.
Steven Cheng-Xian Li, Bo Jiang, and Benjamin Marlin. Misgan: Learning from incomplete data with generative adversarial networks. arXiv preprint arXiv:1902.0995 2019.
Roderick JA Little and Donald B Rubin. Statistical analysis with missing data, volume 793. John Wiley & Sons, 2019.
Xinyue Liu, Chara Aggarwal, Yu-Feng Li, Xiaugnan Kong, Xinyuan Sun, and Saket Sathe. Kernl- ized matrix factorization for collaborative filtering. In Proceedings of the 2016 SIAM International Conference on Data Mining, pp- 378-386, SIAM, 2016.
Chao Ma, Sebastian Tschiatschek, Konstantina Palla, José Miguel Hernández-Lobato Sebastian Nowozin, and Cheng Zhang. Eddi: Efficient dynamic discovery of high-value information with partial vae. arXiv preprint 1809. 142, 2018.
Jinwen Ma, Lei Xu, and Michael Jordan. Asymptotic convergence rate of the em algorithm for gaussian mixtures. Neural Computation, 12(12))2881-200 2000.
Pierre-Alexandh Mattei and Jes Frellsen. Miwae: Deep generative modelling and imputation of incomplete data sets. In International Conference on Machine Learning, pp. 4413-4423, PMLR, 2019.
Xiao-Li Meng. ence, pp. 538-558, 1994.
inferences with uncongenial sources of input. Statistical Sci-
Xiao-Li Meng et al. On the rate of convergence of the eem algorithm. The Annals of Statistics, 22 (1):326-339, 1994.
Danilo Rezende and Shakir Mohamed, Variational inference with normalizing flows. In Interna- tional Conference on Machine Learning, pp. 1530-1538. PMLR, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International conference on machine learning, pp. 1278-1286. PMLR, 2014.
Trevor W Richardson, Wencheng Wu, Lei Lin, Beilei Xu, and Edgar A Bernal. Mcflow: Monte carlo flow models for data imputation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
14205 14205-14214, 2020.
Guido Sanguinetti and Neil D Lawrence. Missing data in kernel pea. In European Conference on Machine Learning, pp. 751-758. Springer, 2006.
Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th international conference on World Wide Web, pp. 285-295, 2001.
Peter Schmitt, Jonas Mandel, and Mickael Guedj. A comparison of six methods for missing data imputation. Journal of Biometrics & Biostatistics, 6(1):1, 2015.
RS Somasundaram and R Nedunchezhian. Evaluation of three simple imputation methods for hancing preprocessing of data with missing values. International Journal of Computer Applica- fions, 14-1) 2011.

---

Under review as a conference paper at ICLR 2022
Daniel Stekhoven and Peter Bühlmann. Missforest con--ammettiic missing value imputation for mixed-type data. Bioinformatics, 28(1):112-113) 2012.
Takeshi Teshima, Isao Ishikawa, Koichi Tojo, Kenta Oono, Masahiro Ikeda, and Masashi Sugiyama. Coupling- based invertible neural networks are universal diffeomorphism approximators. arXiv preprint arXiv:2006. 1469, 2020.
CF Jeff Wu. On the convergence properties of the em algorithm. The Annals of statistics, pp. 95-103, 1983.
Hai xian Wang, Quan bing Zhang, Bin Luo, and Sui Wei. Robust mixture modelling using multivari- ate t-distribution with missing information. Pattern Recognition Letters, 25(6):701-710, 2004.
Junyuan Xic, Linli Xu, and Enhong Chen. Image denoising and inpainting with deep neural net-
works. Advances in neural information processing systems, 25:341-349, 2012.
Raymond Yeh, Chen Chen, Teck Yian Lim, Mark Hasegawa-Johnson, and Minh N Do. Semantic image inpainting with perceptual and contextual losses. arXiv preprint trXiv:1607.075399 2(3), 2016.
Jinsung Yoon, James Jordon, and Mihacla Schaar. Gain: Missing data imputation using generative adversarial nets. International Conference on Machine Learning, pp. 5689-5698. PMLR, 2018.
Adriano Z Zambom and Dias Ronaldo. A review of kernel density estimation with applications to econometrics. International Econometric Review, (2)) 2013.
Ruofei Zhao, Yuanzhi Li, Yuekai Sun, et al. Statistical convergence of the algorithm on gaussian mixture models. Electronic Journal of Statistics, (4(1):632-66,, 2020.
A AFFINE COUPLING LAYERS
The basic idea behind normalizing flow is to find a transformation (of X) that would be able to represent a complex density function as a function of simpler ones, such as a multivariate Gaussian random variable (to be denoted by Z). The quest for an invertible transformation J. such that 2 (X) (0, 7) can be challenging. but in theory such a transformation does exist. Brockwell 2007 showed the existence of such transformation which converts any p-dimensional random variable X to p independent uniform random variables U, = (X) ~ (0, 1) for j = I,..., which are known as universal residuals. Thus, choosing Z, = f, (X) = (x)) ~ (o, 1), where (-) denotes the cumulative distribution function of a standard normal distribution, we can show the existence of Z = f(X), where f=)())= = (x), f,- fx)( that enables to transform X to Z. However, finding such nonlinear transform based on observed data is challenging and so we use sequence of simpler transforms in the spirit of universal residuals, for simplicity and computational efficiency.
Following the framework of Real NVP DDinh et al. 2016), the NF used throughout this work consists of sequence of affine coupling layers fatt ++ y defined
Yi:d = X1:d
(16)
Yd+1:p Xd+1ip exp (s (x1:a)) (x1:d)
where the scale and shift parameters s(-) and t(-) usually implemented by neural networks, and C is the element-wiss wise product.
Therefore, the input split into two parts:the first d dimensions stay the same, while the other dimensions undergo an affine transformation whose parameters are the functions of the first d mensions.
12

---

Under review as a conference paper at ICLR 2022
B ONLINE EM FOR MISSING DATA IMPUTATION
In this section, the steps of vanilla EM in the context of missing data imputation are reviewed. We then show how the online EM framework proposed by Cappé Moulines (2009) can be easily applied here.
B.
VANILLA EM
Given iid data points {z, (z,, []² distributed under (u, E), the E-step of each EM iteration evaluates the conditional expectation of the complete data likelihood:
n Q (0: = Σ
(10;
Ege (log Nog.N(an) (2z;; 2)
(17)
where Φb = (m,, ), and (00) ( (1) are the estimates at the thh iteration. We also follow the treatment by Cappé & Moulines (2009) to normalize the conditional expectation by 1/n for easier transition to the online version of EM.
To calculate conditional expectation inside the summation, use the fact that the conditional distribution of z!" given zद is still Gaussian:
2. [2;; B
~N( _Sm((t)
(18)
where
1 +
()
m;o:
(2000)
2 -p(0
(19)
55m(()
()()
S(t) m.
(t)
(+)
m.m.
a,,,
m.m
m; and o, are the missing and observed masks respectively, and the subscripts of (,( {()) rep- resent the slicing indexes.
Then it's easily easy to find
((()) =
[ (2 14)
)
T E-1
2 11
(20)
+Tr (((D-1) m.m.
5im((t) }]
log 2
where Tr(-) denotes the matrix trace, and 2 is in fact the imputed data vector whose missing part replaced by the conditional mean ç
mm(()
When it comes to the M-step, the derivatives of Q(;: ((1)) are calculated
0Q
Σ
2
11)
0μ
i=1
(21)
0Q
2
11)
(2)0)
14
1
Σ
1
2n
2
where 5:(0) is p x p matrix satisfying (1))
Sm((t) and all other elements equal to 0.
m.m.
Therefore, the maximizer of Q(1; ((1)
13

---

Under review as a conference paper at ICLR 2022
((+11)
i=1
(22)
((+1))
Σ
- (((tt)) (z.0) - -u(tt)² + 50
Here we provide remark that helps to motivate the optimization of EMFlow described in Section
3.4
Remark. Given an incomplete data point "² (H, E), its likelihood maximized at zm E [z")
To prove it, first note that the log-likelihood of z can be written as
1 z pm
T
t pm
log.N22" 20 Σ)
Σ-1
+C
(23)
2
z" Ho
z"
Ho
Denote the conditional mean and covariance of T P Mm Em (Eoo) Σ mm
(z" -Ha) and
Emo (Zoo) Com. The matrix block-wise inversion gives
E mm
Emo Eoo
Σ Σ
Ep En
E
Σ Com
=
(24)
where
Ej Σ
Σ 12 - 5ー
Em 5-1
ΣS = -E-1 00 Emo -1
(25)
En Σmm Σn-Σ
Emoo
5- Em
D
mo Ez1
After series of linear algebra, we have
lzg ,2, ))
(26)
(z μi) T S (z (
(z
(z ma)) +C
As Σー1 is positive definite, obvious that the likelihood reaches the maximum at t P.
B.2
ONLINE EM
It's clear that the EM algorithm needs to process the whole dataset at each iteration, which becomes impractical when dealing with huge datasets. To address this problem, Cappé & Moulines (2009)
propose to replace the reestimation functional in the E-step by a stochastic approximation step:
((+))())= = p++1)((())) + Pt+1
1 B
Σ SEB
E Ten
log ʃ (4; x
(27)
where f is the probability density of complete data, {p are a series of step sizes satisfying the conditions specified in equation B (1...... (1....,n) denotes the indexes of mini-batch of samples, and [B] is the batch size.
Under some mild regulations such the complete data model belongs to exponential family Cappé & Moulines (2009), equation boils down to a stochastic approximation of the estimates suffi- cient statistics:
14

---

Under review as a conference paper at ICLR 2022
1
g((+1) = (1- Po+1) + pr+1
[S(z)|z
(28)
B
BEB
where S(z) is the sufficient statistic of f.
Note that the sufficient statistics for multivariate Gaussian are just sample mean and sample covari- ance. Therefore, equation 8 follows immediately form equation 28
C INTERPRET THE OPTIMIZATION
As described in Section 3.1 the estimated model parameters and the imputed data are supposed to maximize the complete data likelihood (only one data point considered here for illustration purpose), namely,
(8 v. p. Σ)
arg max log px (x(, V.p.
While it's hard to optimize such objective simultaneously. it's possible to optimize it alternatively. For example, learning Li (vb) in equation corresponds to updating i while keeping other variables unchanged.
The operations performed in the latent space needs more explanation. First of all, it shown in appendix B that the imputation increases the complete likelihood in the latent space, i.e.,
N(2: p. Σ) (z: Σ)
(30)
where z and 2 are the embedding vectors before and after the imputation, respectively.
However, the increase of likelihood in the latent space doesn't guarantee the same thing in the data space. Therefore, the flow model updated again by optimizing L2(1b) in equation 14 to ensure that x f (Z) has a high likelihood in the data space X. Note that although x doesn necessarily belong to X', the reconstruction penalty Lree defined in equation 15 forces to be close to X'. Finally, in the re-imputation phase, the imputed data x are updated by projecting x onto X².
D IMPLEMENTATION DETAILS
Model Initialization Each inference iteration consists of training phase and a re-imputation phase. And since the latter phase updates the current imputation, the parameters of the flow model fo are reinitialized after each itcration to learn the new data density faster.
The base distribution estimate A(z; p. Σ) is initialized at the very beginming when the online EM encounters the first batch of embedding vectors. Specifically ç and Σ are initialized to be the sample mean and sample covariance. By default, the base distribution is also reinitialized along with the flow model after each iteration.
where the covariance estimate becomes ill-conditioned during the inference. Such instability can be traced back to two primary sources: (i) the initial naive imputation has really bad quality and the inter-feature dependency is distorted significantly; (ii) the batch size is close to or less than the number features, which makes it hard estimate the covariance directly (e.g. Fan al. 2008 Chen et al.) 2011).
Stabilization of Online EM Online EM be unstable in
Fortunately, those two issues can be solved easily, To reduce the impact of the initial naive imputa- tion and make the covariance estimation more robust, we enlarge the diagonal entries of the original covariance estimate from equation & proportionally:
Rob S + Diag Σ
(31)
15

---

Under review as a conference paper at ICLR 2022
where Diag( Σ) is diagonal matrix with the same diagonal entries as Σ, and 3 positive hyper- parameter. In practice, 3 would be decreased gradually to 0 during the first few iterations as the quality of current imputation gets better and better.
To address the second issuc, the training phase in Algorithm ! can be modified to make the online EM more data points. When updating the base distribution, the current batch of embedding vectors can be concatenated with previous ones to form a super-ubacc with a maximum size Saper- the maximum size is exceeded, the most previous embedding vectors would be excluded. And since the previous batches are already imputed, the super-batch brings nearly no additional computational overhead.
E SIMULATE MAR ON UCI DATASETS
To simulate MAR, the first 70% features of each data point x; is retained, and the remaining 30% features are removed with probability:
10.7pJ Σ 1=1
sigmoid
Zij
(32)
where I-] denotes the integer part of a number.
Note that different data sets would exhibit different missing rates under our MAR setting. Table shows the dimensions of UCI data sets used in our experiment as well as their MAR missing rates for the last 30% features.
Table 4: Information of UCI datasets.
Data
Features
Samples
MAR missing rate
News
60
39644
0.35
Air
13
9357
0.62
Letter
16
19999
0.57
Concrete
9
1030
0.50
Review
24
5454
0,49
Credit
23
30000
0.30
Energy
28
19735
0.39
CTG
21
2126
0.29
Song
90
92743
0.32
Wine
12
6497
0.26
Web
550
58638
0.12
F HYPERPARAMETER ROBUSTNESS
EMFlow has relatively simpler architecture than GAN-based models, and there are just two most important two hyperparameters specifie it. One is the power index y the step size schedule of online EM (see equation 10), while the other is the strength of the reconstruction error in the objective (see equation 14). From our extensive experiments, ? E [0.6, 0.9) often leads to consistent results. On the other hand, a practical guide for choosing is make the magnitudes of first and second term of equation 14 don't differ too much.
Figure 4 shows RMSE the test set of Online News Popularity dataset versus different values of y and a, while all other hyperparameters are kept fixed. The RMSEs all configurations are well below that obtained by MCFlow, and their fluctuations almost negligible.
16

---

Under review as a conference paper at ICLR 2022
0.165
0.165
0.160
0.160
RMSE
0.155
0.155
EMFlow
MCFlow
0.150
0.150
0.145
0.145
0.140
0.140
0.60
0.65
0.70
0.75
0.80
0.85
0.90
10a
10"
107
Y
Figure 4: RMSE between true and imputed values the test set of Online News Popularity dataset with different choices of y and a.
F.1 HYPERPARAMETERS FOR UCI DATASETS
We use 106 and choose Pt 0.99 --0.8 as the step size schedule for all UCI datasets. During the training of EMFlow, the batch size is 256 and the learning rate is 1 x 10 Compared to MCAR, the initial imputation can be more difficult under MAR where the marginal observed density is distorted more. Therefore, the robust covariance estimation in equation 31 is adopted by default under MAR. Specifically, we use 10-2 for the first iterations, iterations, and 0 for the remaining ones.
-10-3 for the next two
F.2 HYPERPARAMETER, FOR IMAGE DATASETS
For experiments with the image datasets, the step size schedule of online EM is again Pt. 0.99- For MNIST and CIFAR-10, choose to be 5 10a and 1 x 10 respectively. During the training of EMFlow, the batch size is 512 and the learning rate 1 x 10 10 Since image datasets have much larger dimensions than UCI datasets, the super-batch approach with size of 3000 is adopted. Additionally, similar robust covariance estimation used by UCI datasets also applied to CIFAR-10.
G ADDITIONAL EXPERIMENT
G.1 MODEL CONGENIALITY
We also consider the congeniality of the imputation model that measures its ability to preserve the feature-label relationship (Meng) 1994 Burgess al. 2013). To quantify how the feature- label relationship is changed after imputation, calculated the euclidean distance between the coefficients two lincar or logistic regression models, one learned from a complete test set and the other learned from the corresponding imputed version. Three datasets with continuous discrete targets were used in our experiments, and the results are shown in Table It shows that the gaps in the measured congeniality actually coincide with those in the measured RMSE.
Table 5: Congeniality of imputation models (lower is better).
Credit
Letter
News
GAIN
1.3310 = 0.1378
4.2957 + 0.2212
0.0575 + 0.0762
MCFlow
1.1080 = 0.1363
3.8157 + 0,2580
0.0541 + 0.0699
EMFlow
0.9141 = 0.1112
3.7611 0.1910
0.0229 + 0.0240
17

---

Under review as a conference paper at ICLR 2022
Table 6: Imputation results of MIWAE UCI datasets RMSE (lower is better).
MCAR
MAR
Data
EMFlow
MIWAR
EMFlow
MIWAF
News
139 + .001
203 + .004
.172 + .000
.195 + .005
Air
.097 + .005
.101 + .008
.040 + .001
.034 + .001
Letter
.111 + .001
.078 + .001
.110 = .001
.085 + .001
Concrete
147 +: .004
.151 +: .009
.133 + .006
.138 + .009
Review
.229 + .003
.234 + .007
.144 = .004
192 = .007
Credit
.125 = .001
.130 + .001
.024 .001
.026 = .002
Energy
.086 + .001
.101 + .011
.15 .002
.67 = .002
CTG
.104 + .006
.102 + .005
105 = .001
.108 = .002
Song
.025 + .000
.031 + .001
.024 + .000
.025 + .000
Wine
.066 + .001
.072 + .002
102 + .002
.899 + .002
Web
.0006 + .0003
.0022 + .0010
.0022
= .0042
.0066 = .0109
G.2 COMPARISON TO MIWAE
In this section, we compare EMFlow with MIWAE that introduces the importance-weighted autoen- coder for imputation tasks. For UCI datasets, follow the implementation provided by the author On the other hand, the implementation of MIWAE on image datasets are not provided, and thus we tried our best to tune MIWAE on MNIST.
As shown in Table EMFlow and MIWAE have mixed performance on UCI datasets, and the differences in most cases are quite small. Table 7 and g shows that EMFlow outperforms MIWAE on MNIST in terms of the RMSE test set and post-imputation accuracy. We also note that EMFlow still converges much faster than MIWAE
Table 7: Imputation results MIWAE on MNIST
RMSE (lower is better)
Missing Rate
.1
.2
.3
.4
.5
.6
.7
.8
.9
MIWAE
.1022
.1057
.1120
.1185
.1194
.1233
.1409
.1672
.2275
MNIST
EMFlow
.0726
.0775
.0832
.0901
.0986
.1100
.1260
.1504
.1951
Table 8: Classification accuracy of MIWAE on imputed MNIST (higher better)
Missing Rate
.1
.2
.3
.4
.7
.8
.9
MIWAE
9792
.9878
.9633
9672
.521
9469
.9237
9331
.7005
MNIST
EMFlow
.9894
.884
.9822
.9788
.9860
.9824
.9696
.9253
.7502
G.3 EFFECT OF INITIAL IMPUTATION
EMFlow is an iterative imputation framework, and the nearest-neighhor (NN) imputation is per- formed for image datasets at the beginning as warm-start in all the previous experiments. In this section, we compare NN imputation and zero imputation as the starting point of EMFlow and shows the convergence of the training loss, the RMSE on the test set, and the Frobenius norm of the co- variance matrix in the latent space. As shown in Figure, 5] the difference between NN imputation and zero imputation is negligible and the convergence under both cases is fast.
https:://githuu..comm//// wae
18

---

Under review as a conference paper at ICLR 2022
putatin Imputation
Imputation Imputation
1002
0:332
Training LoSs
Frobenius Norm
12.03
RMSF
400
0:000
6300
NN Imputation
Zero Imp station
-1722
Epoch
Epoch
Epech
putation Imputation
0300
Imputation Zero Imputation
Imputation
0325
37.5
Zero imputation
Frobenius Norm
Train rg LO55
-1224
0200
-0022
0333
Epoch
Epoch
Epoch
Figure 5: The convergence of training loss, test set RMSE and Frobenius norm the latent covari- ance matrix using NN or zero imputation the the start on MNIST. The first and second rows of plots correspond to the missing rates of 0.5 and 0.9.
19


Reviews:
[{"review_id": "WvlKhCyqo-e", "reviewer": "Reviewer_zdAh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The study is well designed. however, there are few limitations that needs to be acknowledged.\n\nThe data structure is using imaging data as example. This kind of dataset having high neighborhood correlation and imputation using EM with latent space will be less challenging comparing to time series dataset which also have additional layers of correlation between time points. Perhaps need to show the performance on those datasets.\n\nThe author did not show the performance in MNAR. Therefore the novelty remains limited.\n\nThe author also need to realize the limitation for all EM method comparing to FCS multiple imputation which considering the uncertainty of the imputed value.", "summary_of_the_paper": "The authors propose a novel architecture EMFlow for missing data imputation. The authors also show the results of various experiments with multivariate and image datasets. Finally, the authors report the accuracy of post-imputation classification on image datasets.", "summary_of_the_review": "The study and the expected results are acceptable; however, this framework was tested on imaging dataset, which has unique characteristics. for example, one can not assume similar performance when the method is tested on longitudinal clinical data, unless tested systematically on actual data. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "P0QsASnSkq", "reviewer": "Reviewer_Twou", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Although the imputation of MCAR and MAR scenarios has been studied for decades and in theory is not a problem, its applications in high-dimensional and real-world data with complicated distributions can be interesting from a practical perspective.\n\nThe work adapts the online EM algorithm for the missing data imputation. I quickly go through the online EM extension, which seems sound and no problem.\n\nHowever, I have some concerns regarding the assumptions of NF.\nIt would be necessary to justify the assumptions, especially, why the two assumptions are reasonable to believe.\n\nFirstly, inter-feature dependencies in the observation space are different from the inter-feature dependencies in the latent space. According to the studies in factor analysis and nonlinear ICA, the latent representation is not identifiable. So it is not straightforward to believe that the dependencies in the latent space are consistent with the dependencies in the observation space. Then, why would it be reasonable to do imputation in the latent space?\n\nSecondly, note that according to the property of the change of variable formulation used by NF, the neighbors in the latent/source variable space are not necessary to be the neighbors in the observation space.\nAnother fact about NF is that even though the covariance matrix of NF is an identity matrix, it can still model the data distribution well. Then what are the dependencies modeled by the covariance of the multivariate Gaussian distribution in the latent space? Why we could believe that they are corresponding to inter-feature dependencies in the observed data?\n\nA final minor concern is about the experiments. To have a better view of the comparison and the benefits of the proposed method, more comparison in the experiment section would be required, especially, the works using VAE and the methods without using generative models.\n", "summary_of_the_paper": "The paper aims at imputing missing data which are MCAR and MAR. \nFor modeling the observed data distribution, it utilizes the framework of normalizing flow of which the latent variable/source variable space is Gaussian.\nBy assuming the consistency of inter-feature dependencies in the latent/source variable space, it applies online EM for the imputation of the latent space variables.\nIn the experiments, the proposed method, EMFlow is compared with GAIN, MisGAN, and MCFlow.", "summary_of_the_review": "The work focuses on MCAR and MAR cases and extends NF and online EM for the missing data problem, which can be interesting for applications from a practical perspective. But the assumptions and main idea of the work can be lack justification, i.e., the relationship between the dependencies in the latent space and observation space needs to be elaborated and clarified. Moreover, a more thorough comparison with other related works would be helpful to better evaluate the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8LLp1OznxG", "reviewer": "Reviewer_L7zK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The quality of the paper is generally good. The proposed model combines the online EM and flow models to do imputation, which looks to me a quite reasonable design. The model is evaluated using a number of UCI multi-variable datasets and two image datasets (MNIST and CIFAR-10). The performance is generally better than baselines including MCFlow. Because sampling is avoided by using EM, it converges faster than MCFlow.\n\nThe presentation of the paper is quite clear. It is organized, well-written, and quite easy to follow. However, my main concern is that the novelty of the paper seems to be limited, especially when compared with MCFlow; it seems to me that the major difference is that this paper replaces the sampling with the EM algorithm. It is a very reasonable extension, yet it has already been widely studied in various domains and applications.\n\nThe authors claim in the introduction section that EM can be applied in an interpretable way, which motivates the authors to use EM in this paper, but this point is not further discussed in the paper. I would be very interested to see discussions about how interpretability can be enhanced by this proposed model.\n\nAnother minor typo: in the line below Eq. (2), should it be $\\subseteq$ instead of $\\in$ between $\\mathcal{X}^\\prime_i$ and $\\mathcal{X}$?", "summary_of_the_paper": "This paper presents a model named EMFlow, which performs data imputation in the latent space using the online EM algorithm together with the normalizing flow models. The normalizing flow models aim to capture the complete data density $p_X$ and the bidirectional mapping between the data space and the latent space, even when the data is only partially observed. The parameters in the latent space are updated using an online EM algorithm. Thanks to the feature-wise mapping, the dependency between features in the data space is carried over to the latent space and hence the imputation can be done. Evaluation using ten UCI datasets, MNIST, and CIFAR-10 datasets show impressive improvement against baseline models and the convergence is faster than MCFlow.", "summary_of_the_review": "The presentation is clear, the model is theoretically sound, and experiments show impressive improvement against baseline models for most of the dataset and missing rates. However, the novelty is somewhat limited, especially when compared with MCFlow.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "22dnWja_Rf", "reviewer": "Reviewer_tdhx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "EMFlow builds most closely upon MCFlow, mentioned in the paper. Both models use normalizing flows (NF) to capture the input data distribution, and MCFlow uses a standard MLP to find a latent vector maximizing the log-likelihood of the missing data. EMFlow uses NFs with EM to maximize the probability of the latent vector (under a probabilistic model — a multivariate normal) corresponding to the missing data. EMFlow has strong empirical results compared to the baseline of MCFlow. While the paper does not compare to GAN-based imputation methods, which might outperform simpler architectures, the lack of comparison seems fair given the ease of training EMFlow relative to a GAN-based method. Particularly looking at convergence traces in Figure 2. Overall the text is clear, well-organized, and easy to follow; there are only minor typos.\n\nMain strengths:\nThe architecture presented here is relatively straightforward, and does not have many hyperparameters, leading to relatively easy training and implementation. The choice of a multivariate gaussian latent space distribution leads easy conditioning and marginalization, and thus lends itself well to the online EM algorithm presented for imputation. The empirical results are strong, and the methodology is appealing for the reasons listed above. While not far from MCFlows, it provides enough of a conceptual and performance improvement to be a significant contribution to that work.\n\nMain weaknesses and areas to address:\nThe tasks in the empirical section, while not trivial, are not the most difficult examples. MisGAN, for instance, highlights results on CelebA, a more complicated learning task, as well as a higher-dimensional dataset than any listed in the paper. It might strengthen the paper to present results on a higher-dimensional dataset like CelebA or others. Towards this point, it might also be worth evaluating the FID score of imputed images on a dataset like CelebA, rather than just reporting classification accuracy or RMSE.\n\nThe dimensionality and general dataset descriptions are missing from the text and should be included. Particularly of importance is to include the dimensionality of the various datasets analyzed to support the claim that EMFlow works well for high-dimensional data.\n\nThe nearest neighbor imputation in MNIST and CIFAR-10 to seed EMFLow is quite reasonable. However, this should be compared to as a baseline as well. It is not clear the difference this warm-start makes in practice. Furthermore, EMFlow is able to be seeded with NN imputation, whereas MCFlow is not. EMFlow produces much better looking and smoother images than MCFlow, which contains far more noise, on CIFAR-10, and these differences should be due to algorithm changes in EMFlow and not NN imputation. The concrete suggestion here is to either report NN imputation as a baseline method on it’s own, or seed EMFlow with a random imputation as a form of ablation experiment. A third type of ablation experiment different from the two mentioned can be left up to the authors. Finally, to what degree is the fast convergence a result of the warm-start imputation?\n\nHigh-dimensional image data are thought to lie on lower-dimensional manifolds. It is unlikely that the covariance matrix ($\\Sigma$) learned in Z-space has full rank. As the data dimensionality continues to scale, it seems this methodology of estimating the empirical covariance will likely not scale as well, even with the robust estimator in (31). Could EMFlow be adapted to work with low-rank covariance estimators? Or perhaps the authors could show further analysis of convergence of the covariance matrix estimation during optimization.\n\nCan EMFLow be adapted to impute categorical data?", "summary_of_the_paper": "This paper presents a novel imputation method for high-dimensional datasets that typically serve as benchmarks for machine learning methods. This method (EMFlow) innovates by training a normalizing flow network to map input data samples to a multivariate Gaussian, where imputation is performed via an online version of expectation-maximization (EM), which is commonly used for missing data imputation. EMFlow is applied across regression tasks in datasets in the UCI machine learning dataset repository, and standard image classification datasets, MNIST and CIFAR-10. Empirical results show strong performance for missing data imputation, as well as downstream classification from these imputations, and the model design choices and well-constructed architecture make for easy training and fast optimization convergence.", "summary_of_the_review": "Imputation is a key and general ML problem. EMFlow is an intuitive framework methodologically and in practice seems easy to train and performs well on real data. It is principled in its approach — relying on the rigorous EM algorithm as its base. However, EMFlow (albeit fairly due to training complexity) does not compare to what might be state-of-the-art GAN-based imputation methods. It is also unclear how EMFlow would scale to even higher-dimensional datasets given the modeling assumptions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a data imputation method for MCAR and MAR data by combining EM and normalizing flows.  The paper is clearly written.  The idea is interesting and they show better performance compared to MCFlow and competing methods on ten multivariate UCI data, MNIST and CFAR10 image data.

Issues regarding limited novelty compared to MCFlow was raised.
Issues regarding the validity of Assumption 2 on the dependencies in the latent space and observation space was also raised.
Decision: Reject