Title: LEARNING FROM PARTIALLY-OBSERVED MULTI-MODAL DATA WITH VARIATIONAL AUTOENCODERS

Abstract: Learning from only partially-observed data for imputation has been an active research area. Despite promising progress on unimodal data imputation (e.g., image in-painting), models designed for multimodal data imputation are far from satisfactory. In this paper, we propose variational selective autoencoders (VSAE) for this task. Different from previous works, our proposed VSAE learns only from partially-observed data. VSAE is capable of learning the joint distribution of observed and unobserved modalities as well as the imputation mask, resulting in a unified model for various down-stream tasks including data generation and imputation. Evaluation on both synthetic high-dimensional and challenging lowdimensional multi-modality datasets shows significant improvement over the stateof-the-art data imputation models.

1. INTRODUCTION:

Learning from data is an integral part of machine learning and artificial intelligence. Modern deep learning techniques rely heavily on extracting information form large scale datasets. While such frameworks have been shown to be effective on various down-stream tasks such as classification, regression, representation learning, and prediction, it is typically crucial to have access to clean and complete training data. Complete data in this case can be either labeled data (for classification), or time-series data with no missing values (for regression), or simply image with no missing pixels (for generation). As such, if a model can only access partially-observed data, the performance will likely be much worse than those trained with fully-observed data, if not completely failing. In practical scenarios, however, it is usually costly to acquire clean and complete data due to the limited human resources and time. Having a model designed to learn and extract information from partially-observed data will not only largely increase the application spectrum of deep learning based models, but also provide benefit to new down-stream tasks, for example, data imputation.

Data imputation with deep generative models has been an active research area (Yoon et al., 2018; Ivanov et al., 2019; Nazabal et al., 2018) . Despite promising progress, there are still challenges in learning effective models. First, some prior works focus on learning from fully-observed data while performing imputation on partially-observed data during test phase (Suzuki et al., 2016; Ivanov et al., 2019) . Second, they usually have strong assumptions on missingness mechanism (see A.1) such as data is missing completely at random (MCAR) (Yoon et al., 2018) . Third, mostly unimodal imputation such as image in-painting has been explored for high-dimensional data (Ivanov et al., 2019; Mattei & Frellsen, 2019) . Unimodal data refers to data with only one modality such as image, video, or text. Modeling any combination of data modalities is not well-established yet, which apparently limits the potential of such models, since raw data in real-life is usually acquired in a multimodal manner (Ngiam et al., 2011) with more than one source of data gathered to represent a practical scenario. In practice, one or more of the modalities maybe be missing, leading to a challenging multimodal data imputation task.

In this work, we propose Variational Selective Autoencoder (VSAE) for multimodal data generation and imputation. Our proposed VSAE tries to address the challenges above by learning from partiallyobserved training data. By constructing an encoder for each modality independently, the latent representation selectively takes only the observed modalities as input, while a set of decoders maps the latent codes to not only full data (including both observed and unobserved modalities), but also a mask representing the missingness scheme. Thus, it can model the joint distribution of the data and the mask together and avoid limiting assumptions such as MCAR, and is optimized efficiently with a single variational objective. In our experimental validation, we evaluate our proposed VSAE on both synthetic high-dimensional multimodal data and challenging low-dimensional tabular data, and show that VSAE can outperform state-of-the-art baseline models for data imputation task. The contributions are summarized as follows:

(1) A novel framework VSAE to learn from partially-observed multimodal data.

(2) The proposed VSAE is capable of learning the joint distribution of observed and unobserved modalities as well as the imputation mask, resulting in a unified model for various down-stream tasks including data generation and imputation with relaxed assumptions on missigness mechanism.

(3) Evaluation on both synthetic high-dimensional and challenging low-dimensional multimodal datasets shows improvement over the state-of-the-art data imputation models.

2. RELATED WORK:

Our work is related to literature on data imputation and multi-modal representation learning. In this section, we briefly review recent models proposed in these two domains.

Data Imputation. Classical imputation methods such as MICE (Buuren & Groothuis-Oudshoorn, 2010) and MissForest (Stekhoven & Bühlmann, 2011) learn discriminative models to impute missing features from observed ones. With recent advances in deep learning, several deep imputation models have been proposed based on autoencoders (Vincent et al., 2008; Gondara & Wang, 2017; Ivanov et al., 2019) , generative adversarial nets (GANs) (Yoon et al., 2018; Li et al., 2019) , and autoregressive models (Bachman & Precup, 2015) . GAN-based imputation method GAIN proposed by Yoon et al. (2018) assumes that data is missing completely at random. Moreover, this method does not scale to high-dimensional multimodal data. Several VAE based methods (Ivanov et al., 2019; Nazabal et al., 2018; Mattei & Frellsen, 2019) have been proposed in recent years. Ivanov et al. (2019) formulated VAE with arbitrary conditioning (VAEAC) which allows generation of missing data conditioned on any combination of observed data. This algorithm needs complete data during training and cannot learn from partially-observed data only. Nazabal et al. (2018) and Mattei & Frellsen (2019) modified VAE formulation to model the likelihood of the observed data only. However, they require training of a separate generative network for each dimension thereby increasing computational requirements. In contrast, our method aims to model joint distribution of observed and unobserved data along with the missing pattern (imputation mask). This enables our model to perform both data generation and imputation even under relaxed assumptions on missingness mechanism (see Appendix A.1).

Learning from Multimodal Data. A class of prior works such as conditional VAE (Sohn et al., 2015) and conditional multimodal VAE (Pandey & Dukkipati, 2017) focus on learning the conditional likelihood of the modalities. However, these models requires complete data during training and cannot handle arbitrary conditioning. Alternatively, several generative models aim to model joint distribution of all modalities (Ngiam et al., 2011; Srivastava & Salakhutdinov, 2012; Sohn et al., 2014; Suzuki et al., 2016) . However, multimodal VAE based methods such as joint multimodal VAE (Suzuki et al., 2016) and multimodal factorization model (MFM) (Tsai et al., 2019) require complete data during training. On the other hand, Wu & Goodman (2018) proposed another multimodal VAE (namely MVAE) can be trained with incomplete data. This model leverages a shared latent space for all modalities and obtains an approximate joint posterior for the shared space assuming each modalities to be factorized. However, if training data is complete, this model cannot learn the individual inference networks and consequently does not learn to handle missing data during test. Building over multimodal VAE approaches, our model aims to address the shortcomings above within a flexible framework. In particular, our model can learn multimodal representations from partially observed training data and perform data imputation from arbitrary subset of modalities during test. By employing a factorized multimodal representations in the latent space it resembles disentangled models which can train factors specialized for learning from different parts of data (Tsai et al., 2019) .

3. METHOD:

In this section, we introduce a novel VAE-based framework named Variational Selective Autoencoder (VSAE) to learn from partially-observed multimodal data. We first formalize our problem and then provide a detailed description of our model. 3.1 PROBLEM STATEMENT Let x = [x 1 , x 2 ..., x M ] be the complete data with M modalities, where x i denotes the feature representation for the i-th modality. The size of each x i varies and can be very high-dimensional (e.g. multimedia data) or low-dimensional (e.g. tabular data). We define an M -dimensional binary mask variable m ∈ {0, 1} M to represent the observed and unobserved modalities: m i = 1 if the i-th modality is observed and 0 if unobserved. Thus we have the set of observed modalities O = {i|m i = 1}, and the set of unobserved modalities U = {i|m i = 0}. O and U are complementary subsets of all modalities. Accordingly, we denote the representation for the observed and unobserved modalities with x o = [x i |m i = 1] and x u = [x i |m i = 0], respectively. In this paper, we assume the data x and the mask m are dependent, and aim to model the joint distribution of them together.

As a result of such joint modeling, VSAE has higher capacity and can be used for both data imputation and data/mask generation. We encoder the multimodal data to a latent space factorized with respect to the modalities. To handle training and test with partially-observed data, the variational latent variable of each modality is modeled selectively to choose between a unimodal encoder if the corresponding modality is observed, or a multimodal encoder if the modality is unobserved. In addition, all the modalities and mask are reconstructed by decoding the aggregated latent codes through decoders.

3.2 BACKGROUND: VARIATIONAL AUTOENCODER VAE (Kingma & Welling, 2013) is a probabilistic latent variable model to generate a random variable x from a latent variable z with a prior distribution p(z) according to the marginalized distribution p(x) = E z∼p(z) p(x|z) = p(x|z)p(z)dz. However, this is computationally intractable, so the likelihood log p(x) is approximated by variational lower bound (ELBO) L θ,φ (x):

log p(x) ≥ L θ,φ (x) = E z∼q φ (z|x) [log p θ (x|z)] -D KL [q φ (z|x)||p(z)].

(1)

In this equation, q φ (z|x) is a proposal distribution to approximate intractable true posterior p(z|x) and parameterized by an inference network (a.k.a encoder). p θ (x|z) is the conditional likelihood parameterized by another generative network (a.k.a decoder). D KL is the Kullback-Leibler (KL) divergence between the prior and the proposal distribution and functions as a regularizer term,

D KL [q φ (z|x)||p(z)] = E z∼q φ (z|x) [log q φ (z|x) -log p(z)].

To train this model L θ,φ (x) is optimized over all training data with respect to the parameters θ and φ. For more details see Appendix A.2.

3.3. PROPOSED MODEL: VARIATIONAL SELECTIVE AUTOENCODER:

Our goal is to model the joint distribution p(x, m) = p(x, m|z)p(z)dz where

x = [x o , x u ].

Following VAE formulation, we construct a proposal distribution q(z|x, m) to approximate the intractable true posterior. See the architecture in Figure 1 , we denote the parameters of encoder by {φ, ψ}, and decoders of data and mask by θ and respectively. A lower bound of log p(x, m) can be derived as:

L φ,ψ,θ, (x, m) = E z∼q φ,ψ (z|x,m) [log p θ, (x, m|z)] -D KL [q φ,ψ (z|x, m)||p(z)] = E z∼q φ,ψ (z|x,m) [log p θ (x|m, z) + log p (m|z) -log q φ,ψ (z|x, m) + log p(z)].

(2) We assume the variational latent variables can be factorized with respect to the modalities z = [z 1 , z 2 , ..., z M ], which is a standard assumption for multimodal data (Tsai et al., 2019) :

EQUATION

Given this, we define the proposal distribution parameterized by φ and ψ for each modality as

q φ,ψ (z i |x, m) = q φ (z i |x i ) if m i = 1 q ψ (z i |x o , m) if m i = 0 (4)

This is based on the intuitive assumption that the latent space of each modality is independent of other modalities given its data is observed. But, if the data is missing for some modality, its latent space is constructed from the other observed modalities. We call this selective proposal distribution.

In the decoder, the probability distribution also factorizes over the modalities assuming that the reconstructions are conditionally independent given the complete set of latent variables of all modalities:

EQUATION

To summarize, the ELBO in Equation 2 can be rewritten as

EQUATION

where z i ∼ q φ,ψ (z i |x, m) according to the selective proposal distribution given in Equation 4.

For training the model, the ELBO should be maximized over training data. However under partiallyobserved setting, x u is missing and unavailable even during training. Thus, we define the objective function for training by taking expectation over

x u L φ,ψ,θ, (x o , m) = E xu [L φ,ψ,θ, (x o , x u , m)]

(7) Only one term in Equation 6 is dependent on x u , so the final objective function is obtained as

EQUATION

In the proposed algorithm, we approximate E xj [log p θ (x j |m, z)], j ∈ U using reconstructed unobserved data sampling from the prior network. Our experiments show that even a single sample is sufficient to learn the model effectively. In fact, the prior network can be used as a self supervision mechanism to find the most likely samples which dominate the other samples when taking the expectation. In Equation 8, p θ (x i |m, z) is the decoding term of corresponding modality x i and the type of distribution depends on the data. The mask decoding term p θ (m|z) is factorized Bernoulli distribution modeling the binary mask variable. The prior is standard normal distribution

p(z) = M i=1 p(z i ) = M i=1 N (z i ; 0, I) which is fully-factorized.

3.4. NETWORK MODULES:

We construct each module of our model using neural networks and optimize the parameters via backpropagation techniques. Following the terms in standard VAE, VSAE is composed of encoders and decoders. The architecture is shown in Figure 1 . The whole architecture can be viewed as an integration of two auto-encoding structures: the top-branch data-wise encoders/decoders and the bottom-branch mask-wise encoders/decoder. The selective proposal distribution chooses between the unimodal and multimodal encoders, depending on whether the data is observed or not. The outputs of all encoders are sampled and aggregated to provide input to all the decoders. In the rest of this section we explain different modules. See Appendix B for further implementation details.

Selective Factorized Encoders Standard proposal distribution of VAEs depends on the whole data and can not handle incomplete input. To overcome this, we introduce our selective proposal distribution, which is factorized w.r.t the modalities. As defined in Equation 4, the unimodal proposal distribution q φ (z i |x i ) is inferred only from each individual observed modality (modeled by a set of separate encoders parameterized by φ ). If the modality is unobserved, the multimodal proposal distribution q ψ (z i |x o , m) (a single encoder parameterized by ψ) is used to infer corresponding latent variables from other observed modalities and mask. Hence, the learned model can impute the missing information by combining unimodal proposal distribution of observed modalities and multimodal proposal distribution of the unobserved modalities. The condition on the mask could make the model aware of the missing pattern and help attend to observed modalities. We model all the proposal distributions as normal distributions by setting the outputs of all encoders as mean and covariance of a normal distribution. The reparameterization in standard VAE is used for end-to-end training.

Decoding through Latent Variable Aggregator F Selected and sampled from proper proposal distributions for all modalities, the variational latent codes can be fed to the downstream decoders even when the observation is incomplete. To do this, the information from different modalities are combined by aggregating their stochastic latent codes before they are decoded using a decoder:

p (m|z) = p (m|F(z)), p θ (x i |z, m) = p θ (x i |F(z), m)).

Here, we choose the aggregator F(•) = concat(•), i.e., concatenating the latent codes. One may also use other aggregation functions such as max/mean pooling or matrix fusion (Veit et al., 2018) to combine latent codes from all modalities. The decoders take the shared aggregated latent codes as input to generate data and mask.

Mask Vector Encoding and Decoding:

The mask variable m is encoded into the latent space through the multimodal proposal network. The latent space is shared by the mask and data decoders. The mask decoder is parameterized using an MLP in our implementation. We assume each dimension of the mask variable is an independent Bernoulli distribution.

Training With reparameterization trick (Kingma & Welling, 2013), we can jointly optimize the objective derived in Equation 8 with respect to these parameters defined above on training set:

EQUATION

Since Equation 9 only requires the mask and observed data during training, this modified ELBO L φ,θ,ψ, (x o , m) can be optimized without the presence of unobserved modalities. The KLdivergence term is calculated analytically for each factorized term. The conditional log-likelihood term is computed by negating reconstruction loss function. (See Section 4 and Appendix B.2.)

Inference The learned model can be used for both data imputation and generation. For imputation, the observed modalities x o and mask m are fed through the encoders to infer the selective proposal distributions. Then the sampled latent codes are decoded to estimate the unobserved modalities x u . All the modules in Figure 1 are used for imputation. For generation, since no data is available at all, the latent codes are sampled from the prior and go through the decoders to generate the data and the mask. In this way, only modules after the aggregator are used without any inference modules. Table 1 : Feature Imputation on UCI datasets. Missing ratio is 0.5. Categorical and numerical datasets are respectively evaluated by PFC and NRMSE. Last two rows are trained with fully-observed data, potentially serving as an upper bound for imputation models. We show mean and standard deviation over 3 independent runs. For both lower is better.

Figure 2 : Feature Imputations on UCI datasets. Missing ratios (x-axis) are 0.3, 0.5, 0.7. Categorical (top row) and numerical (bottom row) datasets are evaluated by PFC and NRMSE respectively (lower is better for both). We show mean and standard deviation over 3 independent runs.

4. EXPERIMENT:

To demonstrate the effectiveness of our model, we evaluate our model on low-dimensional tabular data imputation and high-dimensional multi-modal data imputation tasks, with extensive comparisons with state-of-the-art deep latent variable models.

Baselines. Prior work on deep latent variable models for data imputation can be categorized into two main classes: (1) models having access to fully-observed data during training, and (2) models only having access to partially observed data during training. In class (1), we report the results of VAEAC (Ivanov et al., 2019) and conditional VAE (Sohn et al., 2015) ; while in class (2), we report results of deterministic Autoencoder (AE), VAE (Kingma & Welling, 2013), conditional VAE (Sohn et al., 2015) (conditioned on mask) and MVAE (Wu & Goodman, 2018) Results and Analysis. Table 1 shows that VSAE outperforms other methods on both numerical and categorical data. The first five rows are trained in partially-observed setting, while the last two trained with fully-observed data. We observe that models trained with partially-observed data can outperform those models trained with fully-observed data on some datasets. We argue this is due to two potential reasons: (1) the mask provides a natural way of dropout on the data space, thereby, helping the model to generalize;

(2) if the data is noisy or has outliers (which is common in low-dimensional data), learning from partially-observed data can improve performance by ignoring these data. However, although our model does not product state-of-the-art results in fully-observed data imputation settings, these models potentially can serve as upper bound if the data is clean.

Figure 2 illustrates that our model generally has lower error with lower variance for all missing ratios. With higher missing ratio (more data is unobserved), our model achieves more stable imputation performance on most of the datasets. On the contrary, there is a performance drop along with higher variance in the case of baselines. We believe this is because of the proposal distribution selection in VSAE. As the missing ratio increases, the input to unimodal encoders stays same while other encoders have to learn to focus on the useful information in data.

High-dimensional Multimodal Data. We synthesize two bimodal datasets using MNIST and SVHN datasets. MNIST contains 28-by-28 gray images (0-9 digits); SVHN contains 32-by-32 RGB images (0-9 digits). We synthesize our datasets by pairing two different digits in MNIST (named MNIST+MNIST) and one digit in MNIST with a same digit in SVHN (named MNIST+SVHN). See Appendix C for more experimental results on multimodal FashionMNIST, MNIST and CMU-MOSI.

Results and Analysis. VSAE has better performance for imputation task on all modalities with lower variance (refer to Table 2 ). Figure 3 presents the qualitative results of imputations on MNIST+MNIST. With masks sampled with different missing ratios, the combined errors on MNIST+MNIST (i.e. sum of MSE in each modality averaged over its dimensions) of our model are 0.1371 ± 0.0001, 0.1376 ± 0.0002 and 0.1379 ± 0.0001 under missing ratio of 0.3, 0.5 and 0.7 (Additional results are in Appendix C.2). This indicates that VSAE is robust under different missing ratios, whereas other baselines are sensitive to the missing ratio. We believe this is because of the underlying mechanism of proper proposal distribution selection. The separate structure of unimodal/multimodal encoders helps VSAE to attend to the observed data. It limits the input of unimodal encoders to observed single modality. Thus it is more robust to the missingness. In contrast, baseline methods have only one single proposal distribution inferred from the whole input. VSAE can easily ignore unobserved noisy modalities and attends on observed useful modalities, while baselines rely on neural networks to learn useful information from the whole data (which is dominated by missing information in case of high missing ratio). For partially-observed training setting, unobserved data is not available even during training. However, the unobserved modality in one data sample could be the observed modality in another data sample. Thus, the multimodal encoders are able to construct the mapping from observable to unobservable information over the whole training set. Multimodal encoders also include the mask vector as input. This allows the multimodal encoders to be aware of the shape of the missingness and forces it to focus on the useful information in the observed modalities. (Mattei & Frellsen, 2019) .

4.3. DATA AND MASK GENERATION:

Unlike conventional methods modeling p(x u |x o ), our method is to model the joint probability p(x o , x u , m). Thus our model can impute missing features and also generate data and mask from scratch. Figure 4 shows the model learns the correlation between different modalities to pair the digits as predefined in the dataset without giving any labels in partially-observed setting.

Our proposed VSAE can also learn to generate mask. The objective ELBO has a mask conditional log-likelihood term. This allows the latent space to have information from mask variables and be able to reconstruct (or generate if sample the prior) the mask vector. In UCI repository experiments, the mask variable follows Bernoulli distribution. After training, we sample from the prior to generate the mask. We calculate the proportion of the unobserved dimensions in generated mask vectors (averaged over 100 samples of the output). Averaged on all datasets, this proportion is 0.3123 ± 0.026, 0.4964 ± 0.005, 0.6927 ± 0.013 for missing ratio of 0.3, 0.5, 0.7. It indicates that our model can learn the mask distribution. We also observe that conditions on the reconstructed mask vector in the data decoders improve the performance. We believe this is because the mask vector can inform the data decoder about the missingness in the data space since the latent space is shared by both all modalities thereby allowing it to generate data from the selective proposal distribution.

5. CONCLUSION:

In this paper, we propose a VAE framework to learn from partially-observed data. Learning from partially-observed data is important but previous deep latent variable models cannot work well on this problem. The proposed model differentiates the observed and unobserved information by selecting a proper proposal distribution. The experimental results show the model can consistently outperform other baselines on low-dimensional tabular data and high-dimensional multimodal data. The model can also generate data with mask directly from prior without any conditions.

A BACKGROUND A.1 IMPUTATION PROCESS AND MISSINGNESS MECHANISMS:

Following (Little & Rubin, 1986) , the imputation process is to learn a generative distribution for unobserved missing data. To be consistent with notations in Section ??, let x = [x 1 , x 2 , ..., x M ] be the complete data of all modalities where x i denote the feature representation for the i-th modality.

We also define m ∈ {0, 1} M as the binary mask vector, where m i = 1 indicates if the i-th modality is observed, and m i = 0 indicates if it is unobsrved:

x ∼ p data (x), m ∼ p(m|x).

(10) Given this, the observed data x o and unobserved data x u are represented accordingly:

x o = [x i |m i = 1], x u = [x i |m i = 0].

(11) In the standard maximum likelihood setting, the unknown parameters are estimated by maximizing the following marginal likelihood, integrating over the unknown missing data values:

p(x o , m) = p(x o , x u , m)dx u = p(x o , x u )p(m|x o , x u )dx u (12)

Little & Rubin (1986) characterizes the missingness mechanism p(m|x o , x u ) in terms of independence relations between the complete data x = [x o , x u ] and the mask m:

• Missing completely at random (MCAR): p(m|x o , x u ) = p(m),

• Missing at random (MAR):

p(m|x o , x u ) = p(m|x o ),

• Not missing at random (NMAR):

p(m|x o , x u ) = p(m|x u ) or p(m|x o , x u ).

Most previous data imputation methods works under MCAR or MAR assumptions since p(x o , m) can be factorized into p(x o )p(m|x o ) or p(x o )p(m). With such decoupling, we do not need missing information to marginalize the likelihood, and it provides a simple but approximate framework to learn from partially-observed data.

A.2 VARIATIONAL AUTOENCODER:

Variational Autoencoder (VAE) (Kingma & Welling, 2013) is a probabilistic generative model, where data is constructed from a latent variable z with a prior distribution p(z). It is composed of an inference network and a generation network to encode and decode data. To model the likelihood of data, the true intractable posterior p(z|x) is approximated by a proposal distribution q φ (z|x), and the whole model is trained until ideally the decoded reconstructions from the latent codes sampled from the approximate posterior match the training data. In the generation module, p θ (x|z), a decoder realized by a deep neural network parameterized by θ, maps a latent variable z to the reconstruction x of observation x. In the inference module, an encoder parameterized by φ produces the sufficient statistics of the approximation posterior q φ (z|x) (a known density family where sampling can be readily done). In vanilla VAE setting, by simplifying approximate posterior as a parameterized diagonal normal distribution and prior as a standard diagonal normal distribution N (0, I), the training criterion is to maximize the following evidence lower bound (ELBO) w.r.t. θ and φ.

EQUATION

) where D KL denotes the Kullback-Leibler (KL) divergence. Usually the prior p(z) and the approximate q φ (z|x) are chosen to be in simple form, such as a Gaussian distribution with diagonal covariance, which allows for an analytic calculation of the KL divergence. While VAE approximates p(x), conditional VAE (Sohn et al., 2015) approximates the conditional distribution p(x|y). By simply introducing a conditional input, CVAE is trained to maximize the following ELBO:

EQUATION

B IMPLEMENTATION DETAILS

B.1 ARCHITECTURE:

In all models, all the layers are modeled by MLP without any skip connections or resnet modules.

Basically, the unimodal encoders take single modality data vector as input to infer the unimodal proposal distribution; the multimodal encoders take the observed data vectors and mask vector as as input to infer the multimodal proposal distributions. The input vector to multimodal encoders should have same length for the neural network. Here we just concatenate all modality vectors and replace the unobserved modality vectors with some noise. In UCI repository experiment, we replace the unobserved modality vectors as standard normal noise. In Bimodal experiment, we simply replace the pixels of unobserved modality as zero. Note that all the baselines has encoders/decoders with same or larger number of parameters than our method. We implement our model using PyTorch.

Unimodal Encoders In UCI repository experiment, the unimodal encoders for numerical data are modeled by 3-layer 64-dim MLPs and the unimodal encoders for categorical data are modeled by 3-layer 64-dim MLPs, all followed by Batch Normalization and Leaky ReLU nonlinear activations. In MNIST+MNIST bimodal experiment, the unimodal encoders are modeled by 3-layer 128-dim MLPs followed by Leaky ReLU nonlinear activations; In MNIST+SVHN bimodal experiment, the unimodal encoders are modeled by 3-layer 512-dim MLPs followed by Leaky ReLU nonlinear activations. We set the latent dimension as 20-dim for every modality in 

Multimodal Encoders:

In general, any model capable of multimodal fusion (Zadeh et al., 2017; Morency et al., 2011) can be used here to map the observed data x o and the mask m to the latent variables z. However, in this paper we simply use an architecture similar to unimodal encoders. The difference is that the input to unimodal encoders are lower dimensional vectors of an individual modalities. But, the input to the multimodal encoders is the complete data vector with unobserved modalities replaced with noise or zeros. As the input to the multimodal encoders is the same for all modalities (i.e., q(z i |x o , m) ∀i), we model the multimodal encoders as one single encoder to take advantage of the parallel matrix calculation speed. Thus the multimodal encoder for every experiment has the same structure as its unmidal encoder but with full-dimensional input.

Aggregator In our models, we simply use vector concatenation as the way of aggregating. In our model, we calculate the conditional log-likelihood of unobserved modality by generating corresponding modalities from prior. We initially train the model for some (empirically we choose 20) epochs without calculating the conditional log-likelihood of x u . And then first feed the partiallyobserved data to the model and generate the unobserved modality xu without calculating any loss; then feed the same batch for another pass, calculate the conditional log-likelihood using real x o and generated x u as ground truth.

B.3 BASELINES:

In our experiments, all the baselines use the same backbone architecture as our model, and the some of the layers are widened to make the total number of parameters same as our proposed model. All baselines for each experiment are trained with same Adam optimizer with same learning rate. All the deep latent variable model baselines have same size of latent variables.

In the setting of AE/VAE, the input is the whole data representation with all the modalties without any mask information; In CVAE w/ mask, the encoder and decoder are both conditioned on the mask vector, while in CVAE w/ data, the observed modalities are fed to encoder and the decoder is conditioned on the observed modalities. VAEAC (Ivanov et al., 2019) is slightly modified to remove all the skip-connections to provide a fair comparison (we do not claim we outperform VAEAC with fully-observed training) and MVAE (Wu & Goodman, 2018) is same as the proposed model architecture. VSAE can jointly model data and mask distribution without any assumption on mask distribution. See A.1 for masking mechanism definitions. Mattei & Frellsen (2019) synthesized the mask from a MAR manner. We similarly follow them to synthesize MAR/NMAR masking mechanism on UCI numerical dataset and compare to state-of-the-art non-MCAR model MIWAE (Mattei & Frellsen, 2019) .

C ADDITIONAL EXPERIMENTAL RESULTS:

Missing At Random (MAR). The mask distribution depends on the observed data. We choose first 25% modalties as default observed data and generate the mask according to the probability that

π(m) = sigmoid( 1 M K k=1 x k )

M is the number of the features and K is the number of default observed features.

Not Missing At Random (NMAR). The mask distribution depends on both observed and unobserved data. We generate the element-wise mask according to the probabilty that π(m i ) = sigmoid(x i ) m i is i-th element in mask vector m of size M .

MCAR:

MAR NMAR MIWAE (Mattei & Frellsen, 2019) 0.467 ± 0.0067 0.493 ± 0.029 0.513 ± 0.035 VSAE(ours) 0.455 ± 0.0003 0.472 ± 0.024 0.455 ± 0.0001

Table 9 : Imputation on MAR/NMAR masking. Missing ratio is based on the values of data following the defined rules above. We show mean and standard deviation over 3 independent runs (lower is better) on Yeast dataset.

C.5 MULTIMODAL EXPERIMENT:

In this section, we include additional experiments on multimodal datasets to demonstrate the general effectiveness of our model. We choose the datasets following MVAE (Wu & Goodman, 2018) and MFM Tsai et al. (2019) .

FashionMNIST We show mean and standard deviation over 3 independent runs (lower is better). ∆ < 0.01.

We choose CMU-MOSI (Zadeh et al., 2016) and ICT-MMMO (Wöllmer et al., 2013) following Tsai et al. (2019) . The author released the features of each modality, and all the numbers are calculated on the feature level. CMU-MOSI (Zadeh et al., 2016) is a collection of 2199 monologue opinion video clips annotated with sentiment. ICT-MMMO (Wöllmer et al., 2013) consists of 340 online social review videos annotated for sentiment. We train all the models using Adam optimizer with learning rate of 1e-3. 

:



Figures and Tables:

Figure 1: Overall architecture. The unimodal proposal network and multimodal proposal network are employed by selection. Modalities are denoted by different colors. Unobserved modalities are shaded. (i.e. blue is observed while red/yellow are unobserved.) The selected variables are indicated by the arrows. Standard normal prior is not plotted for simplicity. All components are trained simultaneously in an end-to-end manner.

Figure 3: Imputation on MNIST+MNIST. Top row visualizes observed modality, middle row unobserved modality, and bottom row shows the imputation of unobserved modality from VSAE.

Figure 4: Generation on MNIST+MNIST. Generated Samples w/o conditional information. As shown, the correspondence between modalities (predefined pairs) are preserved while stochastic multimodal generation.

Figure 5: Imputation on MNIST+MNIST. Top row visualizes observed modality, middle row unobserved modality, and bottom row shows the imputation of unobserved modality from VSAE.

Figure 6: Generation on MNIST+MNIST. Generated Samples w/o conditional information. As shown, the correspondence between modalities (pre-defined pairs) are preserved while generation.

Figure 7: Multiple independent sampling in selected latent space. The leftmost digits are observed images in ground truth, and the right 8 digits are imputations of corresponding unobserved digits.

. Our model VSAE falls in this category since it learns the joint distribution of p(x o , x u , m) given only observed information. Note that class (1) models can empirically represent the upper bound representative capability of imputation models, as they have access to fully-observed data during training. To establish fair comparison, all models in the experiments are implemented with the same backbone structure. Additional information on experimental details can be found in Appendix. B. Imputation on Bimodal datasets.. Missing ratio is 0.5. Last two rows are trained with fully-observed data. We show mean and standard deviation over 3 independent runs (lower is better). ∆ < 0.001. numerical or categorical variables. In our experiments, we randomly sample from independent Bernoulli distributions with pre-defined missing ratio to simulate the masking mechanism. Minmax normalization is then applied to pre-process the numerical data and replace the unobserved dimensions by standard normal noise. We split training/test set by 80%/20% and 20% of training set as validation set to choose the best model. Mean Square Error, Cross Entropy and Binary Cross Entropy are used as reconstruction loss for numerical, categorical and mask variables, respectively. We report the standard measures: NRMSE (i.e. RMSE normalized by the standard deviation of the feature and averaged over all features) for numerical datasets and PFC (i.e. proportion of falsely classified attributes of each feature and averaged over all features) for categorical datasets.

Table content: <table><tr><td/><td colspan="3">MNIST+MNIST(MSE)</td><td colspan="3">MNIST+SVHN(MSE)</td></tr><tr><td/><td colspan="2">MNIST/784 MNIST/784</td><td>combined</td><td>MNIST/784</td><td>SVHN/3072</td><td>combined</td></tr><tr><td>AE</td><td>0.1077 ± ∆</td><td>0.1070 ± ∆</td><td>0.2147 ± ∆</td><td>0.0867 ± ∆</td><td>0.1475 ± ∆</td><td>0.2342 ± ∆</td></tr><tr><td>VAE</td><td>0.0734 ± ∆</td><td>0.0682 ± ∆</td><td>0.1396 ± ∆</td><td colspan="3">0.0714 ± ∆ 0.0559 ± 0.003 0.1273 ± ∆</td></tr><tr><td colspan="2">CVAE w/ mask 0.0733 ± ∆</td><td>0.0679 ± ∆</td><td>0.1412 ± ∆</td><td>0.0692 ± ∆</td><td>0.0558 ± ∆</td><td>0.1251 ± ∆</td></tr><tr><td>MVAE</td><td>0.0760 ± ∆</td><td>0.0802 ± ∆</td><td>0.1562 ± ∆</td><td>0.0707 ± ∆</td><td>0.602 ± ∆</td><td>0.1309 ± ∆</td></tr><tr><td>VSAE (ours)</td><td colspan="3">0.0712 ± ∆ 0.0663 ± ∆ 0.1376 ± ∆</td><td>0.0682 ± ∆</td><td>0.0516 ± ∆</td><td>0.1198 ± ∆</td></tr><tr><td>CVAE w/ data</td><td>0.0694 ± ∆</td><td>0.0646 ± ∆</td><td>0.1340 ± ∆</td><td>0.0716 ± ∆</td><td>0.0550 ± ∆</td><td>0.1266 ± ∆</td></tr><tr><td>VAEAC</td><td>0.0693 ± ∆</td><td>0.0645 ± ∆</td><td>0.1338 ± ∆</td><td>0.0682 ± ∆</td><td>0.0523 ± ∆</td><td>0.1206 ± ∆</td></tr></table>

Imputation

Table content: <table><tr><td>4.2 IMPUTATION ON NON-MCAR MASKING MECHANISMS</td><td/><td/></tr><tr><td/><td>MIWAE</td><td>VSAE</td></tr><tr><td>MCAR</td><td>0.467 ± ∆</td><td>0.455 ± ∆</td></tr><tr><td>MAR</td><td colspan="2">0.493 ± 0.03 0.472 ± 0.02</td></tr><tr><td colspan="2">NMAR 0.513 ± 0.04</td><td>0.456 ± ∆</td></tr></table>

We use Adam optimizer for all models. For UCI numerical experiment, learning rate is 1e-3 and use validation set to find a best model in 1000 epochs. For UCI categorical experiment, learning rate is 1e-2 and use validation set to find a best model in 1000 epochs. For bimodal experiments, learning rate is 1e-4 and use validation set to find a best model in 1000 epochs. All modules in our models are trained jointly.

Table content: <table><tr><td>B.2 TRAINING</td></tr><tr><td>Mask Decoder UCI mask decoder: Linear(20*data-dimension, 64)→ BatchNorm1d(64)→</td></tr><tr><td>LeakyReLU→ Linear(64, 64)→ LeakyReLU→ Linear(64, 64)→ LeakyReLU→ Linear(64, mask-</td></tr><tr><td>dimension)→Sigmoid;</td></tr><tr><td>MNIST+MNIST synthetic mask decoder: Linear(512, 16)→ BatchNorm1d(16)→ LeakyReLU→</td></tr><tr><td>Linear(16,16)→ LeakyReLU→ Linear(16, 16)→ LeakyReLU→ Linear(16, 2)→Sigmoid;</td></tr><tr><td>MNIST+SVHN synthetic mask encoder: Linear(512, 16)→ BatchNorm1d(16)→ LeakyReLU→</td></tr><tr><td>Linear(16,16)→ LeakyReLU→ Linear(16,16)→ LeakyReLU→ Linear(16,2)→Sigmoid;</td></tr><tr><td>Data Decoder As the output is factorized over modalities and for every decoder the input is</td></tr><tr><td>shared as the latent codes sampled from the selective proposal distribution. We implement all</td></tr><tr><td>the decoders of the data modalities as one single decoder for parallel speed. UCI data decoder:</td></tr><tr><td>Linear(20*data-dimension, 128)→ BatchNorm1d(128)→ LeakyReLU→ Linear(128)→ Linear(128,</td></tr><tr><td>128)→ Linear(128, data-dimension);</td></tr><tr><td>MNIST+MNIST synthetic data decoder: Linear(512, 128)→ BatchNorm1d(128)→ LeakyReLU→</td></tr><tr><td>Linear(128,128)→ Linear(128, 128)→ Linear(128, 784)→Sigmoid;</td></tr><tr><td>MNIST+SVHN synthetic mask encoder: Linear(512, 512)→ BatchNorm1d(512)→ LeakyReLU→</td></tr><tr><td>Linear(512,512)→ Linear(512,512)→ Linear(512,784/3072)→Sigmoid;</td></tr></table>

Imputation on Categorical datasets. Missing ratio is 0.5. Last two rows are trained with fully-observed data. Evaluated by PFC, lower is better.

Table content: <table><tr><td colspan="2">C.1 UCI REPOSITORY DATASETS</td><td/><td/></tr><tr><td/><td>Phishing</td><td>Zoo</td><td>Mushroom</td></tr><tr><td>AE</td><td>0.348 ± 0.002</td><td>0.295 ± 0.022</td><td>0.556 ± 0.009</td></tr><tr><td>VAE</td><td>0.293 ± 0.003</td><td>0.304 ± 0.009</td><td>0.470 ± 0.017</td></tr><tr><td colspan="2">CVAE w/ mask 0.241 ± 0.003</td><td>0.270 ± 0.023</td><td>0.445 ± 0.004</td></tr><tr><td>MVAE</td><td>0.308 ± 0.015</td><td>0.233 ± 0.013</td><td>0.586 ± 0.019</td></tr><tr><td>VSAE</td><td colspan="3">0.237 ± 0.001 0.213 ± 0.004 0.396 ± 0.008</td></tr><tr><td>CVAE w/ data</td><td>0.301 ± 0.005</td><td>0.323 ± 0.032</td><td>0.485 ± 0.034</td></tr><tr><td>VAEAC</td><td>0.240 ± 0.006</td><td>0.168 ± 0.006</td><td>0.403 ± 0.006</td></tr></table>

Imputation on Numerical datasets. Missing ratio is 0.5. Last two rows are trained with fully-observed data. Evaluated by NRMSE, lower is better.

Table content: <table><tr><td>C.2 MNIST+MNIST BIMODAL DATASET</td><td/><td/></tr><tr><td>C.2.1 SETUP</td><td/><td/></tr><tr><td colspan="3">MNIST+MNIST bimodal dataset. We randomly pair two digits in MNIST as [0, 9], [1, 8], [2, 7],</td></tr><tr><td colspan="3">[3, 6], [4, 5]. The training/test/validation sets respectively contain 23257/4832/5814 samples.</td></tr><tr><td>C.2.2 ADDITIONAL RESULTS</td><td/><td/></tr><tr><td>0.3</td><td>0.5</td><td>0.7</td></tr></table>

We pair one digit in MNIST with the random same digit in SVHN. The training/test/validation sets respectively contain 44854/10000/11214 samples. For both datasets, we synthesize mask vectors over each modality by sampling from Bernoulli distribution. All mask are fixed after synthesis process. All original data points are only used once. Imputation on MNIST+SVHN. Missing ratio is 0.5. Last two rows are trained with fully-observed data. Evaluated by combined errors of two modalities, lower is better.

Table content: <table><tr><td colspan="2">C.3.2 ADDITIONAL RESULTS</td><td/><td/></tr><tr><td/><td colspan="3">MNIST-MSE/784 SVHN-MSE/3072 Combined Bimodal Error</td></tr><tr><td>AE</td><td>0.0867 ± 0.0001</td><td>0.1475 ± 0.0006</td><td>0.2342 ± 0.0007</td></tr><tr><td>VAE</td><td>0.0714 ± 0.0001</td><td>0.0559 ± 0.0027</td><td>0.1273 ± 0.0003</td></tr><tr><td>CVAE w/ mask</td><td>0.0692 ± 0.0001</td><td>0.0558 ± 0.0003</td><td>0.1251 ± 0.0005</td></tr><tr><td>MVAE</td><td>0.0707 ± 0.0003</td><td>0.602 ± 0.0001</td><td>0.1309 ± 0.0005</td></tr><tr><td>VSAE</td><td colspan="2">0.0682 ± 0.0001 0.0516 ± 0.0001</td><td>0.1198 ± 0.0001</td></tr><tr><td>CVAE w/ data</td><td>0.0716 ± 0.0002</td><td>0.0550 ± 0.0007</td><td>0.1266 ± 0.0005</td></tr><tr><td>VAEAC</td><td>0.0682 ± 0.0001</td><td>0.0523 ± 0.0001</td><td>0.1206 ± 0.0001</td></tr></table>

Imputation on MNIST+SVHN. Missing ratio is 0.3, 0.5 and 0.7. Last two rows are trained with fully-observed data. Evaluated by combined errors of two modalities, lower is better.C.4 IMPUTATION ON NON-MCAR MASKING MECHANISMS

Table content: <table/>

Imputation on Image+Text datasets.. Missing ratio is 0.5. Image and text modality are evaluated by MSE and PFC respectively. Last two rows are trained with fully-observed data.

Table content: <table><tr><td>MNIST</td></tr></table>

Imputation on CMU-MOSI. Missing ratio is 0.5. Last two rows are trained with fullyobserved data. Evaluated by MSE of each modality. We show mean and standard deviation over 3 independent runs (lower is better). ∆ < 0.0005

Table content: <table><tr><td/><td colspan="2">Textual-MSE Acoustic-MSE</td><td>Visual-MSE</td></tr><tr><td>AE</td><td colspan="3">0.035 ± 0.003 0.224 ± 0.025 0.019 ± 0.003</td></tr><tr><td>VAE</td><td>0.034 ± ∆</td><td>0.202 ± ∆</td><td>0.1273 ± ∆</td></tr><tr><td>CVAE w/ mask</td><td>0.43 ± ∆</td><td>0.257 ± 0.002</td><td>0.020 ± ∆</td></tr><tr><td>MVAE</td><td>0.44 ± ∆</td><td>0.213 ± 0.001</td><td>0.025 ± ∆</td></tr><tr><td>VSAE</td><td>0.033 ± ∆</td><td>0.200 ± ∆</td><td>0.017 ± ∆</td></tr><tr><td>CVAE w/ data</td><td>0.036 ± ∆</td><td>0.186 ± ∆</td><td>0.018 ± ∆</td></tr><tr><td>VAEAC</td><td>0.042 ± ∆</td><td>0.257 ± ∆</td><td>0.019 ± ∆</td></tr><tr><td/><td>Acoustic-MSE</td><td>Visual-MSE</td><td>Textual-MSE</td></tr><tr><td>AE</td><td>188.19 ± 2.083</td><td>3.695 ± 0.004</td><td>7.688 ± 0.243</td></tr><tr><td>VAE</td><td>63.26 ± 0.757</td><td>3.676 ± 0.103</td><td>6.153 ± 0.232</td></tr><tr><td>CVAE w/ mask</td><td>61.56 ± 6.584</td><td>3.614 ± 0.015</td><td>6.203 ± 0.423</td></tr><tr><td>MVAE</td><td colspan="3">174.95 ± 117.64 3.569 ± 0.014 8.501 ± 3.561</td></tr><tr><td>VSAE</td><td colspan="3">59.17 ± 4.120 3.569 ± 0.011 5.108 ± 0.003</td></tr><tr><td>CVAE w/ data</td><td>59.22 ± 11.59</td><td>3.367 ± 0.046</td><td>6.398 ± 0.275</td></tr><tr><td>VAEAC</td><td>78.43 ± 8.774</td><td>3.111 ± 0.300</td><td>18.65 ± 0.452</td></tr></table>

Imputation on ICT-MMMO. Missing ratio is 0.5. Last two rows are trained with fullyobserved data. Evaluated by MSE of each modality. We show mean and standard deviation over 3 independent runs (lower is better).

Table content: <table/>

Peer review (iclr2020):
The paper proposed variational selective autoencoders (VSAE) to learn from partially-observed multimodal data. Overall, the proposed method is elegant; however, the presentation, the claim, and the experiments suffer from significant flaws. See below for detailed comments. [Pros] 1. The main idea of the paper is to propose a generative model that can handle partially-observed multimodal data during training. Specifically, prior work considered non-missing data during training, while we can*t always guarantee that all the modalities are available. Especially in the field of multimodal learning, we often face the issue of imperfect sensors. This line of work should be encouraged. 2. In my opinion, the idea is elegant. The way the author handles the missingness is by introducing an auxiliary binary random variable (the mask) for it. Nevertheless, its presentation and Figure 1 makes this elegant idea seems over-complicated. [Cons] 1. [The claim] One of my concerns for this paper is the assumption of the factorized latent variables from multimodal data. Specifically, the author mentioned Tsai et al. assumed factorized latent variables from the multimodal data, while Tsai et al. actually assumed the generation of multimodal data consists of disentangled modality-specific and multimodal factors. It seems to me; the author assumed data from one modality is generated by all the latent factors (see Eq. (11)), then what is the point for assuming the prior of the latent factor is factorized (see Eq. (4) and (5))? One possible explanation is because we want to handle the partially-observable issues from multimodal data, and it would be easier to make the latent factors factorized (see Eq. (6)). The author should comment on this. 2. [Phrasing.] There are too many unconcise or informal phrases in the paper. For example, I don*t understand what does it mean in *However, if training data is complete, ..... handle during missing data during test.* Another example would be the last few paragraphs on page 4; they are very unclear. Also, the author should avoid using the word *simply* too often (see the last few paragraphs on page 5). 3. [Presentation.] The presentation is undesirable. It may make the readers hard to follow the paper. I list some instances here. a. In Eq. (3), it surprises me to see the symbol epsilon without any explanation. b. In Eq. (6), it also surprises me to see no description of phi and psi. The author should also add more explanation here, since Eq. (6) stands a crucial role in the author*s method. c. Figure 1 is over-complicated. d. What is the metric in Table 1 and 2? The author never explains. E.g., link to NRMSE and PFC to the Table. e. What are the two modalities in Table 2? The author should explain. f. The author completely moved the results of MNIST-SVHN to Supplementary. It is fine, but it seems weird that the author still mentioned the setup of MNIST+SVHN in the main text. g. The author mentioned, in Table , the last two rows serve the upper bound for other methods. While some results are even better than the last two rows. The author should explain this. h. Generally speaking, the paper does require a significant effort to polish Section 3 and 4. 4. [Experiments.] The author presented a multimodal representation learning framework for partially-observable multimodal data, while the experiments cannot corraborrate the claim. First, I consider the tabular features as multi-feature data and less to be the multimodal data. Second, the synthetic image pairs are not multimodal in nature. These synthetic setting can be used for sanity check, but cannot be the main part of the experiments. The author can perhaps consider the datasets used by Tsai et al. There are seven datasets, and they can all be modified to the setting of partially-observable multimodal data. Also, since the synthetic image pairs are not multimodal in nature, it is unclear to me for what the messages are conveyed in Figure 3 and 4. I do expect the paper be a strong submission after a significant effort in presentation and experimental designs. Therefore, I vote for weak rejection at this moment.