Learning from Partially-Observed Multimodal Data with Variational Autoencoders
Authors: Yu Gong
Hossein Hajimirsadeghi
Jiawei He
Megha Nawhal
Thibaut Durand
Greg Mori
Venue:  (2020)
Keywords: data imputation
variational autoencoders
generative models

Abstract: Learning from only partially-observed data for imputation has been an active research area. Despite promising progress on unimodal data imputation (e.g., image in-painting), models designed for multimodal data imputation are far from satisfactory. In this paper, we propose variational selective autoencoders (VSAE) for this task. Different from previous works, our proposed VSAE learns only from partially-observed data. The proposed VSAE is capable of learning the joint distribution of observed and unobserved modalities as well as the imputation mask, resulting in a unified model for various down-stream tasks including data generation and imputation.
Evaluation on both synthetic high-dimensional and challenging low-dimensional multi-modality datasets shows significant improvement over the state-of-the-art data imputation models.

Under review as a conference paper at ICLR 2020
LEARNING FROM PARTIALLYY-BSERVEDD MULTI- MODAL DATA WITH VARIATIONAL AUTOENCODERS
Anonymous authors Paper under double-blind review
ABSTRACT
Learning from only partially-observel data for imputation has been an active research area. Despite promising progress on unimodal data imputation (e.g.. image in-painting), models designed for multimodal data imputation are far from satisfactory. In this paper, we propose variational selective autoencoders (VSAE) for this task. Different from previous works, our proposed VSAE learns only from partially-observed data. VSAE is capable of learning the joint distribution of observed and unobserved modalities as well as the imputation mask, resulting in a unified model for various down- down-stream tasks including data generation and imputation. Evaluation both synthetic nigh-dimensional and challenging low- dimensional multi-modality datasets shows significant improvement over the state- of-the-art data imputation models.
INTRODUCTION
Learning from data is an integral part of machine learning and artificial intelligence. Modern deep learning techniques rely heavily on extracting information form large scale datasets. While such frameworks have been shown to be effective various down-stream tasks such classification, regression, representation learning, and prediction, it is typically crucial to have access to clean and complete training data. Complete data in this case can be either labeled data (for classification), or time-series data with no missing values (for regression), or simply image with no missing pixels (for generation). As such, a model can only access partially-observee data, the performance will likely be much worse than those trained with fully-observed data, if not completely failing. In practical scenarios, however, is usually costly to acquire clean and complete data due to the limited human resources and time. Having model designed to learn and extract information from partially data will not only largely increase the application spectrum of deep learning based models, but also provide benefit to new down-stream tasks, for example, data imputation.
Data imputation with deep generative models has been an active research area (Yoon et al., 2018; Ivanov et al., 2019; Nazabal et al., 2018). Despite promising progress, there still challenges in learning effective models. First, some prior works focus on learning from fully-observed data while performing imputation on partially-ohssrred data during et al,, 2019). Second, they usually have strong assumptions on missingness mechanism (see A) such data is missing completely at random (MCAR) (Yoon et al., 2018). Third, mostly unimodal imputation such as image in-painting has been explored for high-dimensional data (Ivanov et al., 2019; Mattei & Frellsen, 2019). Unimodal data refers data with only one modality such as image, video, or text. Modeling any combination of data modalities is not well-established yet, which apparently limits the potential of such models, since raw data in real-life is usually acquired in multimodal manner (Ngiam et al., 201 1) with more than one source of data gathered to represent a practical scenario. In practice, one or more of the modalities maybe be missing, leading to a challenging multimodal data imputation task.
phase (Suzuki et al., 2016; Ivanov
In this work, we propose Variational Selective Autoencoder (VSAE) for multimodal data generation and imputation. Our proposed VSAE tries to address the challenges above by learning from partially- observed training data. By constructing an encoder for each modality independently. the latent representation selectively takes only the observed modalities input, while a set of decoders maps the latent codes to not only full data (including both observed and unobserved modalities), but also mask representing the missingness scheme. Thus, it model the joint distribution of the data

---

Under review as a conference paper at ICLR 2020
and the mask together and avoid limiting assumptions such MCAR, and is optimized efficiently with a single variational objective. In our experimental validation, we evaluate our proposed VSAE on both synthetic high-dimensional multimodal data and challenging low-dimensional tabular data, and show that VSAE can outperform state-of-the-art baseline models for data imputation task. The contributions are summarized as follows:
(I) A novel framework VSAE to learn from partially-observed multimodal data.
(2) The proposed VSAE is capable of learning the joint distribution of observed and unobserved modalities as well as the imputation mask, resulting in unified model for various down-stream tasks including data generation and imputation with relaxed assumptions on missigness mechanism.
(3) Evaluation on both synthetic high-dimensional and challenging low-dimensional multimodal datasets shows improvement over the state-of-thes the-art data imputation models.
RELATED WORK
Our work is related to literature on data imputation and multi-modal representation learning. In this section, we briefly review recent models proposed in these two domains.
Data Imputation. Classical imputation methods such MICE (Buuren & Froothuis-Oudshoon 2010) and MissForest (Stekhoven & Büthlmnn, 2011) 1) learn discriminative models to impute missing features from observed ones. With recent advances in deep learning, several deep imputation models have been proposed based on autoencoders (Vincent et al., 2008; Gondara & Wang, 2017; Ivanov et 2019), generative adversarial nets (GANs) (Yoon et al., 2018; Li et al., 2019), and autoregressive models (Bachman & Precup, 2015). GAN-basee imputation method GAIN proposed by Yoon et al. (2018) assumes that data is missing completely at random. Moreover, this method does not scale to high-dimensional multimodal data. Several VAE based methods (Ivanov et al., 2019; Nazabal et al., 2018; Mattei & Frellsen, 2019) have been proposed in recent years. Ivanov et al. (2019) formulated VAE with arbitrary conditioning (VAEAC) which allows generation of missing data conditioned on any combination of observed data. This algorithm needs complete data during training and canmot learn from partially-observe data only. Nazabal et al. (2018) and Mattei & Frellsen (2019) modified VAE formulation to model the likelihood of the observed data only. However, they require training of separate generative network for each dimension thereby increasing computational requirements. In contrast, our method aims to model joint distribution of observed and unobserved data along with the missing pattern (imputation mask). This enables our model perform both data generation and imputation even under relaxed assumptions on missingness mechanism (see Appendix A.
Learning from Multimodal Data. A class of prior works such as conditional VAE (Sohn et al., 2015) and conditional multimodal VAE (Pandey & Dukkipati, 2017) focus on learning the conditional likelihood of the modalities. However, these models requires complete data during training and cannot handle arbitrary conditioning. Alternatively, several generative models aim to model joint distribution of all modalities (Ngiam et al., 2011: Srivastava Salakhutdinov, 2012; Sohn et al., 2014; Suzuki et al., 2016). However, multimodal VAE based methods such as joint multimodal VAE (Suzuki et al., 2016) and multimodal factorization model (MFM) (Tsai et al., 2019) require complete data during training. On the other hand, Wu & Goodman (2018) proposed another multimodal VAE (namely MVAE) can be trained with incomplete data. This model leverages a shared latent space for all modalities and obtains approximate joint posterior for the shared space assuming each modalities to be factorized. However, if training data is complete, this model cannot learn the individual inference networks and consequently does not learn to handle missing data during test. Building over multimodal VAE approaches, our model aims to address the shortcomings above within a flexible framework. In particular, our model can learn multimodal representations from partially observed training data and perform data imputation from arbitrary subset of modalities during test. By employing factorized multimodal representations in the latent space it resembles disentangled models which can train factors specialized for learning from different parts of data (Tsai et al., 2019).
3 METHOD
In this section, introduce novel VAE-based framework named Variational Selective Autoencoder (VSAE) to learn from partially-observe multimodal data. We first formalize our problem and then provide detailed description of our model.
2

---

Under review as a conference paper at ICLR 2020
data
Unimodal Prepeeal Network
Data Generative Network
Encoder
Decoder
Encoder
Decoder
Encoder
Decoder
observed
Multimodal Preposal Network
Mask Generative Network
Decoder
Encoder
mask
unobserved
variable selection
sampling
aggregator
Figure I: Overall architecture. The unimodal proposal network and multimodal proposal network employed by selection. Modalities denoted by different colors. Unobserved modalities shaded. (i.e. blue observed while red/yellow are unobserved.) The selected variables are indicated by the arrows. Standard normal prior not plotted for simplicity. All components are trained simultaneously in an end-to-end manner.
3.1 PROBLEM STATEMENT
Let [x, X2..., be the complete data with M modalities, where x; denotes the feature representation for the i-th modality. The size of each x; varies and can be very high-dimensional (e.g. multimedia data) low-dimensional (e.g. tabular data). We define M-dimensional binary mask variable m E 13.M to represent the observed and unobserved modalities: m, = if the i-th modality is observed and if unobserved. Thus have the of observed modalities O = [ilm,- = 1}. and the set of unobserved modalities U {ilm: = 0}. O and U are complementary subsets of all modalities. Accordingly, we denote the representation for the observed and unobserved modalities with X. x.|m, 1] and data x and the mask m are dependent, and aim to model the joint distribution of them together.
[x.mm; 0]. respectively. In this paper, assume the
As result of such joint modeling. VSAE has higher capacity and can be used for both data imputation and data/mask generation. encoder the multimodal data to latent space factorized with respect to the modalities. To handle training and test with purtially-observe of each modality is modeled selectively to choose between a unimodal encoder the corresponding modality is observed, modalities and mask are reconstructed by decoding the aggregated latent codes through decoders.
data, the variational latent variable
multimodal encoder if the modality is unobserved. In addition, all the
3.2 BACKGROUND: VARIATIONAL AUTOENCODER
VAE (Kingma & Welling, 2013) is probabilistic latent variable model to generate random variable x from a latent variable z with a prior distribution p(z) according to the marginalized distribution p(x) = (x=x(=)((((z))
(()))))) However, this is computationally intractable, so the
likelihood log is approximated by variational lower bound (ELBO) (x):
logp(x) 2 (x. (logg
D (xa(ux(2(x)) [p(2)).
In this equation, go (z|x) a proposal distribution to approximate intractable true posterior p(z[x) and parameterized by inference network (a.k.a encoder). pe (x|z is the conditional likelihood (KL)
parameterized by another generative network (a.k.a decoder). DKL is the Kullback-Leibh
divergence between the prior and the proposal distribution and functions
regularizer term, Dx1.19. [z(x)][p())] = (xlg)(()))) [log 9s (2)x) logp(z)). To train this model Lo (x) is optimized over all training data with respect the parameters 0 and Φb. For more details Appendix A.2.
3

---

Under review as a conference paper at ICLR 2020
3.3 PROPOSED MODEL: VARIATIONAL SELECTIVE AUTOENCODER
Our goal is to model the joint distribution p(x, m) - p(x, m mzz)p(z)dz where x = [Xo, Xu]- Following VAE formulation, we construct proposal distribution g(z|x, m) approximate the intractable true posterior. See the architecture in Figure 1, we denote the parameters of encoder by for }. and decoders of data and mask by 0 and e respectively. A lower bound of log p(x, m) can be derived as:
C0. m) = =Ex-no v (m/ m)) [log Pe.e e (x, m 2]] D v (z/x, m) [P(()]
-E v (m))x, [log po (x m z) + log Pemmmmmmpppppp log 9s. (z|x, m) + log p(z)].
(2)
We assume the variational latent variables can be factorized with respect the modalities Z1, $2..... ZMI. which is a standard assumption for multimodal data (Tsai et al., 2019):
p(z) II p(z.).
g(z/x, m)
II (2, im
Given this, we define the proposal distribution parameterized by Φ and ai for each modality u [z) x m)
94 (2.(2,)) x,)
if m: =
(4)
Xo. m)
if m4 = 0
This is based on the intuitive assumption that the latent space of each modality is independent of other modalities given its data is observed. But, if the data is missing for some modality, its latent space is constructed from the other observed modalities. We call this selective proposal distribution.
In the decoder, the probability distribution also factorizes over the modalities assuming that the recon- structions are conditionally independent given the complete set of latent variables of all modalities:
log po m, 2) = log po (Xo; Xn m, z)
Σ
log po x; m, 2)
log (xpo(x)/mn (m) (5)
IEO
JEU
To summarize, the ELBO in Equation can be rewritten as
(Xo: Xu, m) =Ez
log pe (x, z) + Σ
Σ
log pe (xy |m, z)
+ Ez leoloogeell (m(|z)]
iEO
jEU
M
Ez, [log q g0. (2: x, m) - log p(:))},
(6)
where z, ~ 20. (2,(x, m) according to the selective proposal distribution given in Equation 4.
For training the model, the ELBO should be maximized over training data. However under partially- observed setting, Xn is missing and unavailable even during training. Thus, we define the objective function for training by taking expectation over Xu
L'ss v. (Xo. m) Ex, Lfo e (Xo, Xu, m)]
Only one term in Equation is dependent on Xu. so the final objective function is obtained as
Co.. m) =E
Σ iEO
log po (x, m, 2) +
Σ jet
Ex, Po m z)] + Eg [log Pe (m/z))
M
Ez, [log do (2 x, m) - log p(z))) where 2; ~ To. (2, x, m)
(8)
In the proposed algorithm, we approximate Ex, [log px (xyIun m,, j E U using reconstructed un- observed data sampling from the prior network. Our experiments show that even a single sample is sufficient to learn the model effectively. In fact, the prior network can be used as self super- vision mechanism to find the most likely samples which dominate the other samples when taking the expectation. In Equation 8, (x; m, 2) is the decoding term of corresponding modality x; and the type of distribution depends on the data. The mask decoding term pe (m|z) is factorized Bernoulli distribution modeling the binary mask variable. The prior is standard normal distribution I (2, 0, which fully-factorized.
p(z) I p(z:)
4

---

Under review as a conference paper at ICLR 2020
3.4 NETWORK MODULES
We construct each module of our model using neural networks and optimize the parameters via backpropagation techniques. Following the terms in standard VAE, VSAE is composed of encoders and decoders. The architecture is shown in Figure 1. The whole architecture can be viewed as an integration of two outo-encoding structures: the top-branch data-wise encoders/decoder: and the bottom-branch mask-wiss encoders/decoder. The selective proposal distribution chooses between the unimodal and multimodal encoders, depending on whether the data is observed or not. The outputs of all encoders are sampled and aggregated to provide input to all the decoders. In the rest of this section we explain different modules. See Appendix B for further implementation details.
Selective Factorized Encoders Standard proposal distribution of VAEs depends on the whole data and can not handle incomplete input. To overcome this, we introduce our selective proposal distribution, which is factorized W.r.t the modalities. As defined Equation 4, the unimodal proposal distribution (x,xx(x;) x;) is inferred only from each individual observed modality (modeled by a set of separate encoders parameterized by Φ ). If the modality is unobserved, the multimodal proposal distribution q (z.|xo. m) (a single encoder parameterized by +)) is used to infer corresponding latent variables from other observed modalities and mask. Hence, the learned model can impute the missing information by combining unimodal proposal distribution of observed modalities and multimodal proposal distribution of the unobserved modalities. The condition on the mask could make the model aware of the missing pattern and help attend to observed modalities. We model all the proposal distributions as normal distributions by setting the outputs of all encoders as mean and covariance of a normal distribution. The reparameterization in standard VAE is used for end-to-end training.
Xoo
Decoding through Latent Variable Aggregator F Selected and sampled from proper proposal distributions for all modalities, the variational latent codes can be fed to the downstream decoders even when the observation is incomplete. To do this, the information from different modalities combined by aggregating their stochastic latent codes before they are decoded using decoder:
Pe Pe(m|z) = Pe (z)). pe (x, zz m) (x F(z). m)). Here, we choose the aggregator F(-) concat(-). i.c., concatenating the latent codes. One may also use other aggregation functions such as max/mean pooling or matrix fusion (Veit et al., 2018 8) to combine latent codes from all modalities.
The decoders take the shared aggregated latent codes as input to generate data and mask.
Mask Vector Encoding and Decoding The mask variable m is encoded into the latent space through the multimodal proposal network. The latent space is shared by the mask and data decoders. The mask decoder e is parameterized using an MLP in our implementation. We assume each dimension of the mask variable an independent Bernoulli distribution.
Training With reparameterization trick (Kingma & Welling, 2013), we can jointly optimize the objective derived in Equation 8 with respect to these parameters defined above training set:
max Ex..m (C''. w.. (Xo: m)]
$.0.4.c
Since Equation 9 only requires the mask and observed data during training, this modified ELBO C' (Xo, m) can be optimized without the presence of unobserved modalities. The KL- divergence term is calculated analytically for each factorized term. The conditional log-likelihood term is computed by negating reconstruction loss function. (See Section 4 and Appendix B.2.)
Inference The learned model can be used for both data imputation and generation. For imputation, the observed modalities X, and mask are fed through the encoders to infer the selective proposal distributions. Then the sampled latent codes are decoded to estimate the unobserved modalities Xu- All the modules in Figure 1 used for imputation. For generation, since no data available at all, the latent codes are sampled from the prior and go through the decoders to generate the data and the mask. In this way, only modules after the aggregator are used without any inference modules.
5

---

Under review as a conference paper at ICLR 2020
Categorical(PFC)
N)MM)))
Phishing
Mushroom
Yeast
Glass
AE
0.348 0.002
0.556 + 0.009
0.737 + 0.036
1.631 = 0.049
VAE
0.2933 + 0.003
0.470 + 0.017
0.468 + 0.003
1.409 = 0.011
CVAB w/ mask
0.2411 0.003
0.445 + 0.004
0.470 + 0.001
1.498 # 01.01
MVAE
0.308 + 0.015
0.586 + 0.019
0.475 + 0.014
1.572 + 01035
VSAE (ours)
0.237 0.001
0.396 - 0.008
0.455 + 0.003
1.312 = 0.021
CVAE w/ data
0.301 + 0.005
0.485 + 0.034
0.449 +: 0.001
1.380 + 0.045
VAEAC
0.240 + 0.006
0.403 + 0.006
0.447 + 0.0016
1.432 + 0.027
Table I: Feature Imputation UCI datasets. Missing ratio is 0.5. Categorical and numerical datasets are respectively evaluated by PFC and NRMSE. Last two rows are trained with oully-observed data, potentially serving as an upper bound for imputation models. We show mean and standard deviation over independent runs. For both lower is better.
Phiniig
Muduroess
0.40
0.35
0.30
0.3
025
608
8.343
0.5
0300
1.3
83
0.3
43
AB VAB XVNB
CVAE
VEAUE
Figure 2: Feature Imputations on UCI datasets. Missing ratios (x-axis) are 0.3, 0.5, 0.7. Categori- cal (top row) and numerical (bottom row) datasets are evaluated by PFC and NRMSE respectively (lower better for both). We show mean and standard deviation over 3 independent runs.
4 EXPERIMENT
To demonstrate the effectiveness of our model, we evaluate our model on low-dimensional tabular data imputation and high-dimensional multi-modal data imputation tasks, with extensive comparisons with state-of-the-art deep latent variable models.
Baselines. Prior work deep latent variable models for data imputation can be categorized into two main classes: (1) models having access to fully-observed data during training, and (2) models only having access to partially observed data during training. class (1), we report the results of VAEAC (Ivanov et al., 2019) and conditional VAE (Sohn et al., 2015); while in class (2), we report results of deterministic Autoencoder (AE), VAE (Kingma & Welling, 2013), conditional VAE (Sohn et al., 2015) (conditioned on mask) and MVAE (Wu & Goodman, 2018). Our model VSAE falls in this category since learns joint distribution of p(Xo; Xu, m) given only observed information. Note that class (1) models can empirically represent the upper bound representative capability of imputation models, as they have access to fully-observed data during training. To establish fair comparison, all models in the experiments are implemented with the same backbone structure. Additional information on experimental details can be found in Appendix. B.
4.1 DATA IMPUTATION
low-dimensional Tabular Data Imputation. the effectiveness of our model tabular data. It contains different tabular datasets with either
We choose UCI repository datasets to demonstrate

---

Under review as a conference paper at ICLR 2020
MNIST-MMNIST(MSSE
MNIST-SVHN(MSE)
MNIST/78 784
MNIST/784
combined
MNIST/784
SVHN/3072
combined
AE
0.1077 + A
0,1070 + A
0.2147 + A
0.0867 A
0.1475 + A
0.2342 + A
VAE
0,0734 =
0.0082 A
0,1396 +
0.0714 = A
0,0559 + 0.003
0.1273 A
CVAE w/ mask
0.0733 = A
0.0679 A
0.1412
0.0692 A
0.0558 +
0.1251 A
MVAF
0.0760 + A
0,0802 A
0.1562 + A
0.0707 A
0.602 - A
0.1309 A
VSAE (ours)
0.0712 A
0.0663 + A
0.1376 + A
0.0682 A
0.0516 + A
0.1198 A
CVAE data VAEAC
0.0694 + A
0.0646 + A
0.1340 + A
0.0716 A
0.0550 + A
0.1266 + A
0.0693 = 0.0645 + A
0,1338 +
0.0682 = A
0.0523 +
0.1206 A
Table 2: Imputation on Bimodal datasets.. Missing ratio is 0.5. Last two rows are trained with fully-observed data. We show mean and standard deviation over 3 independent runs (lower is better). < 0.001.
numerical or categorical variables. In our experiments, we randomly sample from independent Bernouli distributions with pre-defined missing ratio simulate the masking mechanism. Min- max normalization is then applied to pre-process the numerical data and replace the unobserved dimensions by standard normal noise. We split training/test set by 60%/20% and 20% of training set as validation set to choose the best model. Mean Square Error, Cross Entropy and Binary Cross Entropy are used as reconstruction loss for numerical, categorical and mask variables, respectively. We report the standard measures: NRMSE (i.e. RMSE normalized by the standard deviation of the feature and averaged over all features) for numerical datasets and PFC (i.e. proportion of falsely classified attributes of each feature and averaged over all features) for categorical datasets.
Results and Analysis. Table shows that VSAE outperforms other methods on both numerical and categorical data. The first five rows are trained in partially-obeevee setting, while the last two trained with fully-observen data. We observe that models trained with partially-observel data can outperform those models trained with oblly-observed data some datasets. We argue this is due to two potential reasons: (1) the mask provides a natural way of dropout on the data space, thereby, helping the model to generalize; (2) if the data is noisy or has outliers (which is common in low-dimensional data), learning from partially-observed data can improve performance by ignoring these data. However, although our model does not product state-of-the- results in fully-observed data imputation settings. these models potentially can serve as upper bound if the data is clean.
Figure 2 illustrates that our model generally has lower error with lower variance for all missing ratios. With higher missing ratio (more data is unobserved), our model achieves more stable imputation performance on most of the datasets. On the contrary, there is a performance drop along with higher variance in the case of baselines. We believe this is because of the proposal distribution selection in VSAE. As the missing ratio increases, the input to unimodal encoders stays same while other encoders have to learn to focus on the useful information in data.
High-dimensional Multimodal Data. We synthesize two bimodal datasets using MNIST and SVHN datasets. MNIST contains 28-by-28 gray images (0-9 digits); SVHN contains 32-by-32 RGB images (0-9 digits). We synthesize datasets by pairing two different digits in MNIST (named MNIST+MNIST) and one digit in MNIST with a same digit in SVHN (named MNIST+SVHN See Appendix C for more experimental results on multimodal FaShionMNIST, MNIST and CMU-MOSI.
Results and Analysis. VSAE has better performance for imputation task on all modalities with lower variance (refer Table 2). Figure presents the qualitative results of imputations on MNIST-MNIST. With masks sampled with different missing ratios, the combined errors on MNIST+MNIST (i.e. sum of MSE in each modality averaged over its dimensions) of our model are 0.1371 + 0.0001, 0.7376 + 0.0002 and 0379 + 0.0001 under missing ratio of 0.3, 0.5 and 0.7 (Additional results are in Appendix C.2). This indicates that VSAE is robust under different missing ratios, whereas other baselines are sensitive to the missing ratio. We believe this is because of the underlying mechanism of proper proposal distribution selection. The separate structure of unimodal/multimodal encoders helps VSAE to attend to the observed data. It limits the input of unimodal encoders to observed single modality. Thus it is more robust to the missingness. In contrast, baseline methods have only one single proposal distribution inferred from the whole input. VSAE can easily ignore unobserved noisy modalities and attends on observed useful modalities, while baselines rely on neural networks to learn useful information from the whole data (which is dominated by missing information in
7

---

Under review as a conference paper at ICLR 2020
5
4
6
3
0
9
8
1
2
7
4 5
/ 8
2 7
/ 8
0 9
4 5
4/
5
3
6
9
0
1
8
7
2 7
4 5
0 9
4
5
3 6
0 9
4
5
3
6
9
0
/
8 7
2 2
3 6
8
3
6 2
7
4 5
2
7
Figure 3: Imputation on MNIST+MNIST.
Figure 4: Generation on MNIST+MNISST. Gen- erated Samples w/o conditional information. As shown, the correspondence between modalities (pre- defined pairs) are preserved while stochastic multi- modal generation.
Top row visualizes observed modality, mid- dle row unobserved modality, and bottom row shows the imputation of unobserved modality from VSAE.
case of high missing ratio). For partially-observed training setting, unobserved data is not available even during training. However, the unobserved modality in one data sample could be the observed modality in another data sample. Thus, the multimodal encoders are able to construct the mapping from observable to unobservable information over the whole training set. Multimodal encoders also include the mask vector as input. This allows the multimodal encoders to be aware the shape of the missingness and forces it to focus on the useful information in the observed modalities.
4.2 IMPUTATION ON MON-MCAR MASKING MECHANISMS
Sampling mask on predefined missing ratio is MCAR. VSAE can model mask distribution w/o constraints on the masking mechanisms. We also evaluate our model on MAR and NMAR. Mattei & Frellsen (2019) synthesize MAR in a defined rule and we follow them to synthesize both MAR and NMAR (refer to Appendix C.4 for details). Our model can outperform state-of- the-art non-MCAR model MIWAE (Mattei & Frellsen, 2019).
MIWAE
VSAE
MCAR
A
0.455 A
MAR
0.493 + 0.03
0,472 =
NMAR
0.513 0.01
0.456 A
Table 3: Imputation. NRMSE on Yeast. Lower is better. A < 0.01.
4.3 DATA AND MASK GENERATION
Unlike conventional methods modeling p(x,|xx)) our method is to model the joint probability P(Xo: Xu, m). Thus our model can impute missing features and also generate data and mask from scratch. Figure 4 shows the model learns the correlation between different modalities to pair the digits as predefined the dataset without giving any labels in parteall--oberrved setting.
Our proposed VSAE can also learn to generate mask. The objective ELBO has mask conditional log- log-likelihood term. This allows the latent space to have information from mask variables and be able to reconstruct (or generate if sample the prior) the mask vector. In UCI repository experiments, the mask variable follows Bernoulli distribution. After training, we sample from the prior to generate the mask. We calculate the proportion of the unobserved dimensions in generated mask vectors (averaged over 100 samples of the output). Averaged on all datasets, this proportion is 0.3123 + 0.006. 0.4964 + 0.005, 0.6927 + 0.013 for missing ratio of 0.3, 0.5, 0.7. It indicates that our model can learn the mask distribution. We also observe that conditions on the reconstructed mask vector in the data decoders improve the performance. We believe this is because the mask vector inform the data decoder about the missingness in the data space since the latent space is shared by both all modalities thereby allowing it to generate data from the selective proposal distribution.
CONCLUSION
In this paper, propose VAE framework learn from partially-observed data. Learning from partially-observed data is important but previous deep latent variable models cannot work well on this problem. The proposed model differentiates the observed and unobserved information by selecting a proper proposal distribution. The experimental results show the model can consistently outperform other baselines on low-d -i-dimensional tabular data and high-dimensional multimodal data, The model can also generate data with mask directly from prior without any conditions.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Philip Bachman and Doina Precup. Data generation as sequential decision making. Advances in Neural Information Processing Systems, pp. 3249-3257. 2015.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. Interna- tional Conference on Learning Representations, 2016.
S van Buuren and Karin in r. Journal of statistical software, pp. 1-68, 2010.
mice: Multivariate imputation by chained equations
Mare-André Carbonneau, Veronika Cheplygina, Eric Granger, and Ghyslain Gagnon. Multiple instance learning: A survey of problem characteristics and applications. Pattern Recognition, 2018.
Lluis Castrejon, Yusuf Aytar, Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Learning aligned cross-modal representations from weakly aligned data. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2016.
Hong-Min Chu, Chih-Kuan Yeh, and Yu-Chi Yu-Chiang Frank Wang. Deep generative models for weakly- supervised multi-label classification. In The European Conference on Computer Vision (ECCV), September 2018
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 2016.
Lovedeep Gondara and Ke Wang. Multiple imputation using deep denoising autoencoders. arXiv preprint arXiv: ftti:::705.07777 2017.
Jiawei He, Yu Gong, Joseph Marino, Greg Mori, and Andreas Lehrmann. Variational autoencoders with jointly optimized latent dependency structure. In International Conference on Learning Representations, 2019.
Oleg Ivanov, Michael Figurnov, and Dmitry Vetrov. Variational autoencoder with arbitrary condition- ing. In International Conference on Machine Learning, 2019.
Vikas Jain, Nirbhay Modhe, and Piyush Rai. Scalable generative models for multi-label learning with missing labels. In Proceedings of the 34th International Conference Machine Learning. Proceedings Machine Learning Research. PMLR, 2017.
Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to dis- cover cross-domain relations with generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1857-1865. JMLR. org, 2017.
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013,
Diederik P. Kingma, Danilo Jimenez. Rezende, Shakir Mohamed, and Max Welling. Semi-supervised learning with deep generative models. CoRR, foss/1406.529,, 2014.
Steven Cheng-Xian Xian Li, Bo Jiang, and Benjamin Marlin. Misgan: Learning from incomplete data with generative adversarial networks. In International Conference on Learning Representations, 2019.
Roderick A Little and Donald B Rubin. Statistical Analysis with Missing Data. John Wiley & Sons, Inc., New York, NY, USA, 1986. ISBN 0-471-80723 -8225-9.
Plerre-Alexandre Mattei and Jes Frellsen. Miwae:
Deep generative modelling and imputation of incomplete data sets. In International Conference on Machine Learning, pp. 4413-4423, 2019.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3994-4003, 2016.
Louis-Philippe Morency, Rada Mihalcea, and Payal Doshi. Towards multimodal sentiment analysis: Harvesting opinions from the web. In Proceedings of the 13th international conference on multimodal interfaces, pp. 169-176. ACM, 2011.

---

Under review as a conference paper at ICLR 2020
Alfredo Nazabal, Pablo M Olmos, Zoubin Ghahramani, and Isabel Valera. Handling incomplete heterogeneous data using vaes. arXiv preprint arXiv: 2018.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y. Ng. Multi- modal deep learning. In International Conference on Machine Learning, ICML'1I, pp. 689-696, USA, 2011. Omnipress. ISBN 978-1-4503-0619-5.
Gaurav Pandey and Ambedkar Dukkipati. Variational methods for conditional multimodal deep learning. In 2017 International Joint Conference on Neural Networks (IJCNN), pp. 308-315. IEEE, 2017.
Amaia Salvador, Nicholas Hynes, Yusuf Aytar, Javier Marin, Ferda Offi, Ingmar Weber, and Antonio Torralba. Learning cross-modal embeddings for cooking recipes and food images. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), July 2017.
Casper Kaae Sø nderby, Tapani Raiko, Lars Maalø e, Søren Kaae Sø nderby, and Ole Winther. Ladder variational autoencoders. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 3738-3746. Curran Associates, Inc., 2016.
Kihyuk Sohn, Wenling Shang, and Honglak Lee. Improved multimodal deep learning with variation of information. Advances in neural information processing systems, pp. 2141-2149 2014.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems 28, pp. 3483-3491. 2015.
Yuhang Song, Chao Yang, Zhe Lin, Xiaofeng Liu, Qin Huang, Hao Li, and C.-C. Jay Kuo. Contextual- based image inpainting: Infer, match, and translate. In The European Conference on Computer Vision (ECCV), September 2018.
Nitish Srivastava and Ruslan R Salakhutdinov. Multimodal learning with deep boltzmann machines. In Advances neural information processing systems, pp. 1222-2230, 2012.
Daniel Stekhoven and Peter Bühlmann. mixed-type data. Bioinformatics, 28(1):112-113 2011.
missing value imputation for
Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Joint multimodal learning with deep generative models. arXiv preprint /61.089) 1,01891, 2016.
Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdi- nov. Learning factorized multimodal representations. In International Conference on Learning Representations, 2019.
Andreas Veit, Maximilian Nickel, Serge Belongie, and Laurens van der Maaten. Separating self- expression and visual content in hashtag supervision. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In international conference on Machine learning, pp. 1096-10 1903. 103. ACM, 2008
Mike Wu and Noah Goodman. Multimodal generative models for scalable weakly-supervised learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, 2018
M. Wöllmer, F. Weninger, T. Knaup, B. Schuller, C. Sun, Sagae, and L. Morency. Youtube movie reviews: Sentiment analysis in an audio-visual context. IEEE Intelligent Systems, 2013.
Jinsung Yoon, James Jordon, and Mihacla Van Der Schaar. Gain: Missing data imputation using generative adversarial nets. In International Conference on Machine Learning, 2018.
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-PPilipee Morency. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEFE Intelligent Systems, 31(6), 2016.
10

---

Under review as a conference paper at ICLR 2020
Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis Louis-Philippe Morency. Tensor fusion network for multimodal sentiment analysis. Proceedings of the 2017 Conference on Em- pirical Methods in Natural Language Processing, pp. 1103-1114. Association for Computational Linguistics, 2017. doi: 10. 10.18653/1/DD1--1111
Amir Zadeh, Yao-Chong Lim, Paul Pu Liang, and Louis-Philippe Morency. Variational auto-decoder:
Neural generative modeling from partial data. arXiv preprint arXiv:I903.00844 2019.
Ying Zhang and Huchuan Lu. Deep modal projection learning for image-text matching. The European Conference on Computer Vision (ECCV), September 2018.
Yu Zhang, Ying Wei, and Qiang Yang. Learning multitask. In Advances in Neural Information Processing Systems, pp. 5771-5782, 2018.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, 2017.

---

Under review as a conference paper at ICLR 2020
A BACKGROUND
A.1 IMPUTATION PROCESS AND MISSINGNESS MECHANISMS
Following (Little & Rubin, 1986), the imputation process is to learn generative distribution for unobserved missing data. To be consistent with notations in Section ??, let x = [x1, X2,... XMM be the complete data of all modalities where x; denote the feature representation for the i-th modality. We also define E {0. 1 as the binary mask vector, where m. 1 indicates if the i-th modality observed, and m, = 0 indicates if unobsrved:
X ~ Pdsta(
p(m xx..
(10)
Given this, the observed data x, and unobserved data Xu are represented accordingly:
X = | = 1], Xu [x: mm;; = 0]. In the standard maximum likelihood setting, the unknown parameters are estimated by maximizing the following marginal likelihood, integrating over the unknown missing data values:
(11)
P m) - f p(xo: Xu, m)dx_ / p(xo: Xu (u)p(mm/xxx))) Xu)
(12)
Little & Rubin (1986) characterizes the missingness mechanism p(m Xu) in terms of indepen- dence relations between the complete data x [xo= x., and the mask m:
Missing completely random (MCAR): (m Xx: xn) p(m).
Missing at random (MAR): [(m[x., x..) = pmm|xx).
Not missing at random (NMAR): pmm xu) = p(m/xu) or p(mlxx.
Most previous data imputation methods works under MCAR or MAR assumptions since p(x.. m) can be factorized into p(x.)) ) (m)) or p(xx)p(m)) With such decoupling, we do not need missing information to marginalize the likelihood, and it provides a simple but approximate framework to leam from partially- data.
A.2 VARIATIONAL. AUTOENCODER
Variational Autoencoder (VAE) (Kingma & Welling, 2013) is probabilistic generative model, where data is constructed from latent variable z with a prior distribution p(z). It is composed of an inference network and generation network to encode and decode data. To model the likelihood of data, the true intractable posterior p(z[x) is approximated by a proposal distribution go (z(x), and the whole model is trained until ideally the decoded reconstructions from the latent codes sampled from the approximate posterior match the training data. the generation module, Po (x|z), a decoder realized by a deep neural network parameterized by 0, maps latent variable x of observation x. In the inference module, an encoder parameterized by & produces the sufficient statisties of the approximation posterior 4p (z|x) known density family where sampling can be readily done). In vanilla VAE setting, by simplifying approximate posterior as a parameterized diagonal normal distribution and prior as a standard diagonal normal distribution (O, I). the training criterion is to maximize the following evidence lower bound (ELBO) w.r.t. 0 and Φ.
the reconstruction
(z)x) where DxL. denotes the Bullback-Leible (KL) divergence. Usually the prior p(z) and the approximate 9 (z)x) are chosen be in simple form, such as Gaussian distribution with diagonal covariance, which allows for analytic calculation of the KL divergence. While VAE approximates p(x),
Dog((x)= 2 (x) = (2)x) [log po (x (xxx) [z) DRL. [ (z) x [))((())))
(13)
conditional VAE (Sohn et al., 2015) approximates the conditional distribution p(xly). ) By simply introducing conditional input, CVAE is trained to maximize the following ELBO:
log p(xly) 2 (x.ow E. (z)x.)) [log Po (x|z, y) DxL. 44 (z) (2)x,,x,/xx0. x))x
(14)
12

---

Under review as a conference paper at ICLR 2020
B IMPLEMENTATION DETAILS
B.I ARCHITECTURE
In all models, all the layers are modeled by MLP without any skip connections or resnet modules. Basically, the unimodal encoders take single modality data vector input infer the unimodal proposal distribution; the multimodal encoders take the observed data vectors and mask vector as as input to infer the multimodal proposal distributions. The input vector multimodal encoders should have same length for the neural network. Here we just concatenate all modality vectors and replace the unobserved modality vectors with some noise. In UCI repository experiment, we replace the unobserved modality vectors as standard normal noise. In Bimodal experiment, we simply replace the pixels of unobserved modality as zero. Note that all the baselines has encoders/decoders with same or larger number of parameters than our method. We implement our model using Py
Unimodal Encoders In UCI repository experiment, the unimodal encoders for numerical data are modeled by 3-layer 64-dim MLPs and the unimodal encoders for categorical data are modeled by 3-layer 64-dim MLPs, all followed by Batch Normalization and Leaky ReLU nonlinear activations. In MNIST+MNIST bimodal experiment, the unimodal encoders are modeled by 3-layer 128-dim MLPs followed by Leaky ReLU nonlinear activations; In MNIST+ bimodal experiment, the unimodal encoders are modeled by 3-layer 512-dim MLPs followed by Leaky ReLU nonlinear activations. We set the latent dimension 20-dim for every modality in UCI repository experiments and 256-dim for every modality in Bimodal experiments.
UCI data unimodal encoder: Linear(1, 64) BatchNorrm Leaky Linear(64, 64) LeakyRel.U Linear(64, 64)-> LeakyRel.U-> Linear(64, 20);
MNIST+MNIST synthetic unimodal encoder: Linear(data--dmmeniion 128) LeakyReL.U>> Linear(128, 128) LeakyReL.U-> Linear(125 128) Leaky Linear(128, 256);
MNIST+SVHN synthetic unimodal encoder: Linear(dataaddimesiion 512)+ LeakyReLU~> Linear(512,5122 12) LeakyReLU-> 111e 512) LeakyReLU-+ Linear(512, 256);
Multimodal Encoders In general, any model capable of multimodal fusion (Zadeh et al., 2017; Morency al., 2011) can be used here to map the observed data x, and the mask to the latent variables z. However, in this paper we simply use an architecture similar to unimodal encoders. The difference is that the input to unimodal encoders are lower dimensional vectors of individual modalities. But, the input to the multimodal encoders is the complete data vector with unobserved modalities replaced with noise or zeros. As the input the multimodal encoders the same for all modalities (i.e., (2, m) Vi), we model the multimodal encoders as one single encoder to take advantage of the parallel matrix calculation speed. Thus the multimodal encoder for every experiment has the same structure as its unmidal encoder but with full-dimensional input.
Aggregator In our models, we simply use vector concatenation as the way of aggregating.
Mask Decoder UCI mask decoder: Linear(20+Annaddimmenso. 64) BatchNormld(64) LeakyReL.U-+ Linear(64, 64)- Leaky Line-r(64,, 64) Leaky Linear(64,, mask- dimension)--Sigmolli
MNIST+MNIST synthetic mask decoder: Linear(512, 16) BatchNorm (tt.hoooommmm1111-1 LeakyReLU~> Linear( 16) LeakyRll. Linear(16, 1,, 16) Leaky Linear( 16, 2) 2)-Sigmoid;;
MNIST+SVHN synthetic mask encoder: Linear(512,, 16) BatchNorm I d(16))+ LeakyReLUU>> Linear(16,16) LeakyReLU Linear(16,16)-- Leaky Linear( -iggmoio
Data Decoder As the output is factorized over modalities and for every decoder the input is shared as the latent codes sampled from the selective proposal distribution. We implement all the decoders of the data modalities as one single decoder for parallel speed. UCI data decoder: 128) BatchNorm 128) Linear( 128. data-dimension);
Leaky Linear( Linear( (228,
MNIST+MIIST synthetic data decoder: Linear(512, 128) BatchNormld(129)- LeakyReLUU> Leakk Linear( 128,128)~> Linear( 128, 128) Linear( (22,, (84)moS;
MNIST+SVHN synthetic mask encoder: Linear(512, 512) - BatchNormll((12) LeakyReLU~>
Linear(512,5122- Linear(512,5122)- (inn5))))nnnner(5112222222. Linear(5112,784/3077
>Sigmoid;
13

---

Under review as a conference paper at ICLR 2020
B.2 TRAINING
We use Adam optimizer for all models. For UCI numerical experiment, learning rate is le-3 and use validation set to find best model in 1000 epochs. For UCI categorical experiment, learning rate is le-2 and use validation set to find best model in 1000 epochs. For bimodal experiments, learning rate is le-4 and use validation set find a best model in 1000 epochs. All modules our models are trained jointly.
In our model, calculate the conditional log-likelihood of unobserved modality by generating corresponding modalities from prior. We initially train the model for some (empirically we choose 20) epochs without calculating the conditional log likkliihooddd of Xu. And then first feed the partially- observed data the model and generate the unobserved modality Xn, without calculating any loss; then feed the same batch for another pass, calculate the conditional log-likelihood using real X and generated Xu ground truth.
B.3 BASELINES
In our experiments, all the baselines use the same backbone architecture as our model, and the some of the layers are widened to make the total number parameters same as our proposed model. All baselines for each experiment are trained with same Adam optimizer with same learning rate. All the deep latent variable model baselines have same size of latent variables.
In the setting of AE/VAE, the input is the whole data representation with all the modalties without any mask information; In CVAE w/ mask, the encoder and decoder are both conditioned on the mask vector, while in CVAE w/ data, the observed modalities are fed to encoder and the decoder is conditioned on the observed modalities. VAEAC (Ivanov et al., 2019) is slightly modified to remove all the skip-connections to provide a fair comparison (we do not claim we outperform VAEAC with fully-observed training) and MVAE (Wu & Goodman, 2018) is same as the proposed model architecture.
C ADDITIONAL EXPERIMENTAL RESULTS
C.I UCI REPOSITORY DATASETS
Phishing
Zoo
Mushroom
AE
0.348 + 0.002
0.295 + 0.022
0.556 + 0.009
VAE
0.293 + 0.003
0.304
0.009
0.470 + 0.017
CVAE w/ mask
0.241 + 0.003
0.270
+ 0.023
0.445 + 0.004
MVAE
0.308 + 0.015
0.233
0.013
0.586 + 0.019
VSAE
0.237 = 0.001
0.213 + 0.004
0.396 = 0.008
CVAE w/ data
0.301 + 0.005
0.323 + 0.032
0.485 + 0.034
VAEAC
0.240 + 0.006
0.168 + 0.006
0.403 + 0.006
Table 4: Imputation on Categorical datasets. Missing ratio is 0.5. Last two rows are trained with fully-observed data. Evaluated by PFC, lower better.
14

---

Under review as a conference paper at ICLR 2020
Yeast
White Wine
Glass
AE
0.737 = 0.036
0.3772 + 0.0008
1.651 0.049
VAE
0.468 + 0.003
0.3771 + 0.0001
1.409 + 0.011
CVAE w/ mask
0.470 + 0.001
0.3716 0.0001
1.498 + 0.0013
MVAE
0.475 0.014
0.3722 + 0.0009
1.572 0.035
VSAE
0.455 0.003
0.3711 + 0.0002
1.312 = 0.021
CVAE w/ data
0.449 = 0.0001
0.3567 + 0.0016
1.380 = 0.045
VAEAC
0.447 + 0.0016
0.3647 0.0039
1.432 0.027
Table 5: Imputation on Numerical datasets. Missing ratio is 0.5. Last two rows are trained with fully-observed data. Evaluated by NRMSE, lower is better.
C.2 MNIST+MNIST BIMODAL DATASET
C.2.1 SETUP
MNIST+MNIST bimodal dataset.
We randomly pair two digits in MNIST as [0, 9], [1. 8). [2, 7].
[3, 6]. [4, 5]. The training/tes/vaaliattinn sets respectively contain 23257/4832/5814 samples.
C.2.2 ADDITIONAL. RESULTS
0.3
0.5
0.7
AE
0.2124 = 0.0012
0.2147 + 0.0008
0.2180 + 0.0008
VAE
0.1396 = 0.0002
0.1416
0.0001
0.1435
0.0006
CVAE w/ mask
0.1393 = 0.0002
0.1412 # 0.0006
0.1425 + 0.0012
MVAE
0.1547 + 0.0012
0.1562 + 0.0003
0.1579
0.0006
VSAE
0.1371 + 0.0001
0.1376 + 0.0002
0.1379 + 0.0001
CVAE w/ data
0.1336 0.0003
0.1340 0.0003
0.1343 0.0002
VAEAC
0.1333 + 0.0004
0.1338 + 0.0003
0.1344 + 0.0001
Table 6: Imputation MNIST+MNIST. Missing ratio is 0.3, 0.5 and 0.7. Last two
trained with obsly-o data. Evaluated by combined errors of two modalities, lower is better.
5
4
6
3
0
9
8
2
7
4
7
6
0
1
5
9 7
6
9
7
4
5
3
6
9
0
8
7
2
5
2
3
9
8
4
2
0
3 0
2
4
5
3
6
9
0
/
8
7
2
5
7
3
9
8
4
2 O
3
0
Figure Imputation on MNIST+MNIST Top row visualizes observed modality, middle row unobserved modality, and bottom row shows the imputation of unobserved modality from VSAE.
15

---

Under review as a conference paper at ICLR 2020
4
5
/ 8
2 7
/ 8
0 9
4 5
2 7
3 6
3 6
2 7
4 5
0 9
4 5
3 6
0 9
3 6
1 8
4 5
3 6
/ 8
3 6
2 7
4 5
2 7
4 5
O 9
2 7
0 9
4 5
0 9
5 4
4 5
/ 8
0 6
4 5
/ 8
/ 8
2
7
2
7
2 7
8
0
7
3
6
2
7
0
9
Figure 6: Generation on MNIST+MNIST. Generated Samples w/o conditional information. As shown, the correspondence between modalities (pre-defined pairs) preserved while generation.
2
7
7
7
7
7
0
9
9
9
9
9
9
9 9
1
6
8
8
8
8
8
8
8
4
5 5
5 5
5 5
5
3
6
6
6
6
6
6
6
6
Figure 7: Multiple independent sampling in selected latent space. The leftmost digits are observed images in ground truth, and the right 8 digits are imputations of corresponding unobserved digits.
C.3 MNIST+SVHN BIMODAL DATASET
C.3.1 SETUP
MNIST+SVHN bimodal dataset: We pair one digit in MNIST with the random same digit in SVHN. The training/test/aaliattinn sets respectively contain 44854/10000/1 1214 samples. For both datasets, we synthesize mask vectors over each modality by sampling from Bernoulli distribution. All mask are fixed after synthesis process. All original data points are only used once.
C.3.2 ADDITIONAL RESULTS
MNIST-MSE/784
SVHN-MSE/3072
Combined Bimodal Error
AE
0.0867 + 0.0001
0.1475 + 0.0006
0.2342 + 0.0007
VAE
0.0714 = 0.0001
0.0559 = 0.0027
0.1273 0.0003
CVAE w/ mask
0.0692 = 0.0001
0.0558 : 0.0003
0.1251 + 0.0005
MVAE
0.0707 = 0.0003
0.602 + 0.0001
0.1309 0.0005
VSAE
0.0682 + 0.0001
0.0516 + 0.0001
0.1198 + 0.0001
CVAE w/ data VAEAC
0.0716 + 0.0002
0.0550 + 0.0007
0.1266 + 0.0005
0.0682 = 0.0001
0.0523 = 0.0001
0.1206 + 0.0001
Table 7: Imputation on MNIST+SVHN, Missing ratio is 0.5. Last two rows are trained with fully-observed data. Evaluated by combined errors of modalities, lower is better.
16

---

Under review as a conference paper at ICLR 2020
0.3
0.5
0.7
AE
0.1941 + 0.0006
0.2342 + 0.0007
0.2678
0.0012
VAE
0.1264 + 0.0001
0.1273 + 0.0003
0.1322 + 0.0005
CVAE w/ mask
0.1255 0.0002
0.1251
0.0005
0.1295 0.0006
MVAE
0.1275 + 0.0029
0.1309
0.0005
0.1313 + 0.0013
VSAE
0.1217 0.0002
0.1198
0.0001
0.1202
0.0002
CVAR w/ data
0.1288 + 0.0011
0.1266
0.0005
0.1248 0.0003
VAEAC
0.1218 + 0.0002
0.1206 + 0.0001
0.1211 + 0.0001
Table 8: Imputation on MNIST+SVHN. Missing ratio is 0.3, 0.5 and 0.7. Last two rows are trained with fully-observed data. Evaluated by combined errors of two modalities, lower is better.
C.4 IMPUTATION ON NON-MCAR MASKING MECHANISMS
VSAE can jointly model data and mask distribution without any assumption on mask distribution. See A.I for masking mechanism definitions. Mattei & Frellsen (2019) synthesized the mask from MAR manner. We similarly follow them synthesize MAR/NMAR masking mechanism on UCI numerical dataset and compare to state-of- ttate-of-the-art non-MCAR model MIWAE (Mattei & Frellsen, 2019).
Missing At Random (MAR). The mask distribution depends on the observed data. We choose first 25% modalties as default observed data and generate the mask according to the probability that
π (m) sigmoid
x
M
M is the number of the features and K is the number of default observed features.
Not Missing At Random (NMAR). The mask distribution depends on both observed and unob- served data. We generate the element wise mask according to the probabilty that
n (m+) sigmoid(x;)
m; is i-th element in mask vector m size M.
MCAR
MAR
NMAR
MIWAE(Mattei Frellsen, 2019) VSAE(ours)
0.467 + 0.0067
0.493
0.029
0.513 + 0.035
0.455 = 0.0003
0.472 + 0.024
0.455 = 0.0001
Table 9: Imputation MARNMAR masking. Missing ratio is based on the values of data following the defined rules above. We show mean and standard deviation over 3 independent runs (lower better) on Yeast dataset.
C.5 MULTIMODAL EXPERIMENT
In this section, we include additional experiments on multimodal datasets to demonstrate the general effectiveness of our model. We choose the datasets following MVAE (Wu & Goodman, 2018) and MFM Tsai et al. (2019).
17

---

Under review as a conference paper at ICLR 2020
FashionMNIST
MNIST
image (MSE)
text (PFC)
image (MSE)
text (PFC)
AB
86.63 + 1.09
0.366 + A
54.90 + 0.01
0.406 + A
VAE
69.38 0.10
0.411 + A
53.82 0.12
0.406 0.01
CVAE w/ mask
69.53 + 0.65
0.412 + A
53.82 + A
0.419 + A
MVAE
109.95 + 20.78
0.374 =: 0.07
178.40 + 14.29
0.448 + A
VSAE (ours)
68.49 0.19
0.356
53.42 0.05
0.397 0.01
CVAE w/ data
54.15 + 0.03
0,259 + A
47.38 + A
0.237 + A
VAEAC
61.59 + 0.03
0.283 + A
51.49 + 0.06
0.250 + A
Table 10: Imputation Image+Teext datasets.. Missing ratio is 0,5. Image and text modality evaluated by MSE and PFC respectively. Last two rows are trained with fully-observed data. We show mean and standard deviation over 3 independent runs (lower better). A 0.01.
We choose CMU-MOSI (Zadeh et al., 2016) and ICT-MMMO (Wöllmer et al., 2013) following Tsai al. (2019). The author released the features of each modality, and all the numbers are calculated on the feature level. CMU-MOSI (Zadeh et al., 2016) is collection of 2199 monologue opinion video clips annotated with sentiment. ICT-MMMO (Wöllmer et al., 2013) consists of 340 online social review videos annotated for sentiment. We train all the models using Adam optimizer with learning rate le-3.
Textual-MSE
Acoustic-MSE
Visual-MSE
AE
0.035 = 0.003
0,224 + 0.025
0.019 + 0.003
VAE
0.034 + A
0.202 + A
0.1273 +
CVAE w/ mask
0.43 + A
0.257 + 0.002
0.020 + A
MVAE
0.44 +
0.213 0.001
0.025 +
VSAE
0.033 + A
0.200 + A
0.017 A
CVAE w/ data
0.036 + A
0.186 A
0.018 A
VAEAC
0.042 + A
0.257 + A
0.019 + A
Table 11: Imputation on CMU-MOSI Missing ratio is 0.5. Last two rows are trained with fully- observed data. Evaluated by MSE of each modality. We show mean and standard deviation over independent runs (lower is better). A 0.0005
Acoustic-Mel
Visual-MSE
Textual-MSE
AE
188.19 = 2.083
3.695 + 0.004
7.688 0.243
VAE
63.26 + 0.757
3.676 0.103
0.103
6.153 + 0.232
CVAE w/ mask
61.56 = 6.584
3.614 0.015
6.208 0.423
MVAE
174.95 + 117.64
3.569 = 0.014
8.501 + 3.561
VSAE
59.17 + 4.120
3.569 + 0.011
5.108 + 0.003
CVAE w/ data VAEAC
59.22 + 11.59
3.367 + 0.046
6.398 + 0.275
78.43 + 8.774
3.111 +-2 0.300
18.65 + 0.452
Table 12: Imputation on ICT-MMMO. Missing ratio is 0.5. Last two rows are trained with fully- observed data. Evaluated by MSE of each modality. We show mean and standard deviation over independent runs (lower better).
18


Reviews:
[{"review_id": "r1gk4ck25B", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes variational selective autoencoders (VSAE) to learn the joint distribution model of full data (both observed and unobserved modalities) and the mask information from arbitrary partial-observation data. To infer latent variables from partial-observation data, they introduce the selective proposal distribution that switches encoders depending on whether each input modality is observed.\n\nThis paper is well written, and the method proposed in this paper is nice. In particular, the idea of the selective proposal distribution is interesting and provides an effective solution to deal with the problem of missing modality in conventional multimodal learning. The experiment is also well structured and shows higher performance than the existing models.  However, I have some questions and comments, so I’d like you to answer them.\n\nComments:\n- The authors state that x_j is sampled from the \"prior network\" to calculate E_x_j in Equation 10, but I didn’t understand how this network is set up. Could you explain it in detail?\n- The authors claim that adding p(m|z) to the objective function (i.e., generating m from the decoder) allows the latent variable to have mask information. However, I don’t know how effective this is in practice. Specifically, how performance differs compared to when p (m | z) is not used and the decoder p (x | z, m) is conditioned by the mask included in the training set instead of the generated mask?\n- Why did you not do image inpainting in higher-dimensional experiments like Ivanov et al. (2019), i.e., considering each pixel as a different modality? Of course, I know that Ivanov et al. require the full data as input during training, but I’m interested in whether VSAE can perform inpainting properly even if trained given imperfect images.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "H1ly2z2RFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes to impute multimodal data when certain modalities are present. The authors present a variational selective autoencoder model that learns only from partially-observed data. VSAE is capable of learning the joint\ndistribution of observed and unobserved modalities as well as the imputation mask, resulting in a model that is suitable for various down-stream tasks including data generation and imputation. The authors evaluate on both synthetic high-dimensional and challenging low-dimensional multimodal datasets and show improvement over the state-of-the-art data imputation models.\n\nStrengths:\n- This is an interesting paper that is well written and motivated.\n- The authors show good results on several multimodal datasets, improving upon several recent works in learning from missing multimodal data.\n\nWeaknesses:\n- How multimodal are the datasets provided by UCI? It seems like they consist of different tabular datasets with numerical or categorical variables, but it was not clear what the modalities are (each variable is a modality?) and how correlated the modalities are. If they are not correlated at all and share no joint information I'm not sure how these experiments can represent multimodal data. \n- Some of the datasets the authors currently test on are quite toy, especially for the image-based MNIST and SVHN datasets. They should consider larger-scale datasets including image and text-based like VQA/VCR, or video-based like the datasets in (Tsai et al., ICLR 2019).\n- In terms of prediction performance, the authors should also compare to [1] and [2] which either predict the other modalities completely during training or use tensor-based methods to learn from noisy or missing time-series data.\n- One drawback is that this method requires the mask during training. How can it be adapted for scenarios where the mask is not present? In other words, we only see multiple modalities as input, but we are not sure which are noisy and which are not?\n\n[1] Pham et al. Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities, AAAI 2019 \n[2] Liang et al. Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization, ACL 2019\n\n### Post rebuttal ###\nThank you for your detailed answers to my questions.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1eOXTuD5B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a novel training method for variational autoencoders that allows using partially-observed data with multiple modalities. A modality can be a whole block of features (e.g., a MNIST image) or just a single scalar feature. The probabilistic model contains a latent vector per modality. The key idea is to use two types of encoder networks: a unimodal encoder for every modality which is used when the modality is observed, and a shared multimodal encoder that is provided all the observed modalities and produces the latent vectors for the unobserved modalities. The whole latent vector is passed through a decoder that predicts the mask of observed modalities, and another decoder that predicts the actual values of all modalities. The “ground truth” values for the unobserved modalities are provided by sampling from the corresponding latent variables from the prior distribution once at some point of training.\n\nWhile I like the premise of the paper, I feel that it needs more work. My main concern is that sampling the target values for the unobserved modalities from the prior would almost necessarily lead to blurry synthetic “ground truth” for these modalities, which in turn means that the model would produce underconfident predictions for them. The samples from MNIST in Figure 3 are indeed very blurry, supporting this. Furthermore, the claims of the model working for non-MCAR missingness are not substantiated by the experiments. I believe that the paper should currently be rejected, but I encourage the authors to revise the paper.\n\nPros:\n* Generative modelling of partially observed data is a very important topic that would benefit from fresh ideas and new approaches\n* I really like the idea of explicitly modelling the mask/missingness vector. I agree with the authors that this should help a lot with non completely random missingness.\n\nCons:\n* The text is quite hard to read. There are many typos (see below). The text is over the 8 page limit, but I don’t think this is justified. For example, the paragraph around Eqn. (11) just says that the decoder takes in a concatenated latent vector. The MNIST+SVHN dataset setup is described in detail, yet there is no summary of the experimental results, which are presented in the appendix.\n* The approach taken to train on partially-observed data is described in three sentences after the Eqn. (10). The non-observed dimensions are imputed by reconstructions from the prior from a partially trained model. I think that this is the crux of the paper that should be significantly expanded and experimentally validated. It is possible that due to this design choice the method would not produce sharper reconstructions than the original samples from the prior. Figures 3, 5 and 6 indeed show very blurry samples from the model. Furthermore, it is not obvious to me why these prior samples would be sensible at all, given that all modalities have independent latents by construction.\n* The paper states multiple times that VAEAC [Ivanov et al., 2019] cannot handle partially missing data, but I don’t think this is true, since their missing features imputation experiment uses the setup of 50% truly missing features. The trick they use is adding “synthetic” missing features in addition to the real ones and only train on those. See Section 4.3.3 of that paper for more details.\n* The paper states that “it can model the joint distribution of the data and the mask together and avoid limiting assumptions such as MCAR”. However, all experiments only show results in the MCAR setting, so the claim is not experimentally validated.\n* The baselines in the experiments could be improved. First of all, the setup for the AE and VAE is not specified. Secondly, it would be good to include a GAN-based baseline such as GAIN, as well as some more classic feature imputation method, e.g. MICE or MissForest.\n* The experiments do not demonstrate that the model learns a meaningful *conditional* distribution for the missing modalities, since the provided figures show just one sample per conditioning image.\n\nQuestions to the authors:\n1. Could you comment on the differences in your setup in Section 4.1 compared to the VAEAC paper? I’ve noticed that the results you report for this method significantly differ from the original paper, e.g. for VAEAC on Phishing dataset you report PFC of 0.24, whereas the original paper reports 0.394; for Mushroom it’s 0.403 vs. 0.244. I’ve compared the experimental details yet couldn’t find any differences, for example the missing rate is 0.5 in both papers.\n2. How do you explain that all methods have NRMSE > 1 on the Glass dataset (Table 1), meaning that they all most likely perform worse than a constant baseline?\n\nTypos and minor comments:\n* Contributions (1) and (2) should be merged together.\n* Page 2: to literature -> to the literature\n* Page 2: “This algorithm needs complete data during training cannot learn from partially-observed data only.”\n* Equations (1, 2): z and \\phi are not consistently boldfaced\n* Equations (4, 5): you can save some space by only specifying the factorization (left column) and merging the two equations on one row\n* Page 4, bottom: use Bernoulli distribution -> use factorized/independent Bernoulli distribution\n* Page 5, bottom: the word “simply” is used twice\n* Page 9: learn to useful -> learn useful\n* Page 9: term is included -> term included\n* Page 9: variable follows Bernoulli -> variable following Bernoulli\n* Page 9: conditions on -> conditioning on", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJlU6IBK9S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed variational selective autoencoders (VSAE) to learn from partially-observed multimodal data.  Overall, the proposed method is elegant; however, the presentation, the claim, and the experiments suffer from significant flaws. See below for detailed comments.\n\n[Pros]\n1. The main idea of the paper is to propose a generative model that can handle partially-observed multimodal data during training. Specifically, prior work considered non-missing data during training, while we can't always guarantee that all the modalities are available. Especially in the field of multimodal learning, we often face the issue of imperfect sensors. This line of work should be encouraged. \n\n2. In my opinion, the idea is elegant. The way the author handles the missingness is by introducing an auxiliary binary random variable (the mask) for it. Nevertheless, its presentation and Figure 1 makes this elegant idea seems over-complicated.\n[Cons]\n\n1. [The claim] One of my concerns for this paper is the assumption of the factorized latent variables from multimodal data. Specifically, the author mentioned Tsai et al. assumed factorized latent variables from the multimodal data, while Tsai et al. actually assumed the generation of multimodal data consists of disentangled modality-specific and multimodal factors. It seems to me; the author assumed data from one modality is generated by all the latent factors (see Eq. (11)), then what is the point for assuming the prior of the latent factor is factorized (see Eq. (4) and (5))? One possible explanation is because we want to handle the partially-observable issues from multimodal data, and it would be easier to make the latent factors factorized (see Eq. (6)). The author should comment on this. \n\n2. [Phrasing.] There are too many unconcise or informal phrases in the paper. For example, I don't understand what does it mean in \"However, if training data is complete, ..... handle during missing data during test.\" Another example would be the last few paragraphs on page 4; they are very unclear. Also, the author should avoid using the word \"simply\" too often (see the last few paragraphs on page 5). \n\n3. [Presentation.] The presentation is undesirable. It may make the readers hard to follow the paper. I list some instances here. \n\t\ta. In Eq. (3), it surprises me to see the symbol \\epsilon without any explanation. \n\t\tb. In Eq. (6), it also surprises me to see no description of \\phi and \\psi. The author should also add more explanation here, since Eq. (6)  stands a crucial role in the author's method.\n\t\tc. Figure 1 is over-complicated.\n\t\td. What is the metric in Table 1 and 2?  The author never explains. E.g., link to NRMSE and PFC to the Table. \n\t\te. What are the two modalities in Table 2? The author should explain.\n\t\tf. The author completely moved the results of MNIST-SVHN to Supplementary. It is fine, but it seems weird that the author still mentioned the setup of MNIST+SVHN in the main text. \n\t\tg. The author mentioned, in Table , the last two rows serve the upper bound for other methods. While some results are even better than the last two rows. The author should explain this.\n\t\th. Generally speaking, the paper does require a significant effort to polish Section 3 and 4. \n\n4. [Experiments.] The author presented a multimodal representation learning framework for partially-observable multimodal data, while the experiments cannot corraborrate the claim. First, I consider the tabular features as multi-feature data and less to be the multimodal data. Second, the synthetic image pairs are not multimodal in nature. These synthetic setting can be used for sanity check, but cannot be the main part of the experiments. The author can perhaps consider the datasets used by Tsai et al. There are seven datasets, and they can all be modified to the setting of partially-observable multimodal data. Also, since the synthetic image pairs are not multimodal in nature, it is unclear to me for what the messages are conveyed in Figure 3 and 4. \n\n\nI do expect the paper be a strong submission after a significant effort in presentation and experimental designs. Therefore, I vote for weak rejection at this moment.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission proposes a VAE-based method for jointly inferring latent variables and data generation. The method learns from partially-observed multimodal data.

Strengths:
-Learning to generate from partially-observed data is an important and challenging problem.
-The proposed idea is novel and promising.

Weaknesses:
-Some experimental protocols are not fully explained.
-The experiments are not sufficiently comprehensive (comparisons to key baselines are missing).
-More analysis of some surprising results is needed.
-The presentation has much to improve.

The method is promising but the mentioned weaknesses were not sufficiently addressed during discussion. AC agrees with the majority recommendation to reject.

Decision: Reject