Title: JOINT MULTIMODAL LEARNING WITH DEEP GENERA-TIVE MODELS

Abstract: We investigate deep generative models that can exchange multiple modalities bidirectionally, e.g., generating images from corresponding texts and vice versa. Recently, some studies handle multiple modalities on deep generative models, such as variational autoencoders (VAEs). However, these models typically assume that modalities are forced to have a conditioned relation, i.e., we can only generate modalities in one direction. To achieve our objective, we should extract a joint representation that captures high-level concepts among all modalities and through which we can exchange them bi-directionally. As described herein, we propose a joint multimodal variational autoencoder (JMVAE), in which all modalities are independently conditioned on joint representation. In other words, it models a joint distribution of modalities. Furthermore, to be able to generate missing modalities from the remaining modalities properly, we develop an additional method, JMVAE-kl, that is trained by reducing the divergence between JMVAE's encoder and prepared networks of respective modalities. Our experiments show that our proposed method can obtain appropriate joint representation from multiple modalities and that it can generate and reconstruct them more properly than conventional VAEs. We further demonstrate that JMVAE can generate multiple modalities bidirectionally.

1. INTRODUCTION:

In our world, information is represented through various modalities. While images are represented by pixel information, these can also be described with text or tag information. People often exchange such information bi-directionally. For instance, we can not only imagine what "a young female with a smile who does not wear glasses" looks like, but also add this caption to a corresponding photograph. To do so, it is important to extract a joint representation that captures high-level concepts among all modalities. Then we can bi-directionally generate modalities through the joint representations. However, each modality typically has a different kind of dimension and structure, e.g., images (real-valued and dense) and texts (discrete and sparse). Therefore, the relations between each modality and the joint representations might become high nonlinearity. To discover such relations, deep neural network architectures have been used widely for multimodal learning (Ngiam et al., 2011; Srivastava & Salakhutdinov, 2012) . The common approach with these models to learn joint representations is to share the top of hidden layers in modality specific networks. Among them, generative approaches using deep Boltzmann machines (DBMs) (Srivastava & Salakhutdinov, 2012; Sohn et al., 2014) offer the important advantage that these can generate modalities bi-directionally.

Recently, variational autoencoders (VAEs) (Kingma & Welling, 2013; Rezende et al., 2014) have been proposed to estimate flexible deep generative models by variational inference methods. These models use back-propagation during training, so that it can be trained on large-scale and highdimensional dataset compared with DBMs with MCMC training. Some studies have addressed to handle such large-scale and high-dimensional modalities on VAEs, but they are forced to model conditional distribution (Kingma et al., 2014; Sohn et al., 2015; Pandey & Dukkipati, 2016) . Therefore, it can only generate modalities in one direction. For example, we cannot obtain generated images from texts if we train the likelihood of texts given images. To generate modalities bi-directionally,

Reconstruct images:



Generate attributes:

Male : -1 Eyeglasses : -1 Smiling : 1 . .

Reconstruct images with changed attributes:

Not Smiling Male Eyeglasses Not Young

Generate images:

Random faces Average face Input image

Attributes:



JMVAE:

Figure 1 : Various images and attributes generated from an input image. We used the CelebA dataset (Liu et al., 2015) to train and test models in this example. Each yellow box corresponds to different processes. All processes are estimated from a single generative model: the joint multimodal variational autoencoder (JMVAE), which is our proposed model. all modalities should be treated equally under the learned joint representations, which is the same as previous multimodal learning models before VAEs.

As described in this paper, we develop a novel multimodal learning model with VAEs, which we call a joint multimodal variational autoencoder (JMVAE). The most significant feature of our model is that all modalities, x and w (e.g., images and texts), are conditioned independently on a latent variable z corresponding to joint representation, i.e., the JMVAE models a joint distribution of all modalities, p(x, w). Therefore, we can extract a high-level representation that contains all information of modalities. Moreover, since it models a joint distribution, we can draw samples from both p(x|w) and p(w|x). Because, at this time, modalities that we want to generate are usually missing, the inferred latent variable becomes incomplete and generated samples might be collapsed in the testing time when missing modalities are high-dimensional and complicated. To prevent this issue, we propose a method of preparing the new encoders for each modality, p(z|x) and p(z|w), and reducing the divergence between the multimodal encoder p(z|x, w), which we call JMVAE-kl. This contributes to more effective bi-directional generation of modalities, e.g., from face images to texts (attributes) and vice versa (see Figure 1 ).

The main contributions of this paper are as follows:

• We introduce a joint multimodal variational autoencoder (JMVAE), which is the first study to train joint distribution of modalities with VAEs. • We propose an additional method (JMVAE-kl), which prevents generated samples from being collapsed when some modalities are missing. We experimentally confirm that this method solves this issue. 

2. RELATED WORK:

The common approach of multimodal learning with deep neural networks is to share the top of hidden layers in modality specific networks. Ngiam et al. (2011) proposed this approach with deep autoencoders (AEs) and found that it can extract better representations than single modality settings. Srivastava & Salakhutdinov (2012) also took this idea but used deep Boltzmann machines (DBMs) (Salakhutdinov & Hinton, 2009) . DBMs are generative models with undirected connections based on maximum joint likelihood learning of all modalities. Therefore, this model can generate modalities bi-directionally. Sohn et al. (2014) improved this model to exchange multiple modalities effectively, which are based on minimizing the variation of information and JMVAE-kl in ours can be regarded as minimizing it with variational learning on parameterized distributions (see Section 3.3 and Appendix A). However, it is computationally difficult for DBMs to train high-dimensional data such as natural images because of MCMC training.

Recently, VAEs (Kingma & Welling, 2013; Rezende et al., 2014) are used to train such highdimensional modalities. Kingma et al. (2014) ; Sohn et al. (2015) propose conditional VAEs (CVAEs), which maximize a conditional log-likelihood by variational methods. Many studies are based on CVAEs to train various multiple modalities such as handwriting digits and labels (Kingma et al., 2014; Sohn et al., 2015) , object images and degrees of rotation (Kulkarni et al., 2015) , face images and attributes (Larsen et al., 2015; Yan et al., 2015) , and natural images and captions (Mansimov et al., 2015) . The main features of CVAEs are that the relation between modalities is one-way and a latent variable does not contain the information of a conditioned modality1 , which are unsuitable for our objective.

Pandey & Dukkipati (2016) proposed a conditional multimodal autoencoder (CMMA), which also maximizes the conditional log-likelihood. The difference between CVAEs is that a latent variable is connected directly from a conditional variable, i.e., these variables are not independent. Moreover, this model forces the latent representation from an input to be close to the joint representation from multiple inputs, which is similar to JMVAE-kl. However, the CMMA still considers that modalities are generated in fixed direction. This is the most different part from ours.

3. METHODS:

This section first introduces the algorithm of VAEs briefly and then proposes a novel multimodal learning model with VAEs, which we call the joint multimodal variational autoencoder (JMVAE).

3.1. VARIATIONAL AUTOENCODERS:

Given observation variables x and corresponding latent variables z, their generating processes are definable as z ∼ p(z) = N (0, I) and x ∼ p θ (x|z), where θ is the model parameter of p. The objective of VAEs is maximization of the marginal distribution p(x) = p θ (x|z)p(z)dx. Because this distribution is intractable, we instead train the model to maximize the following lower bound of the marginal distribution L V AE (x) as

EQUATION

where q φ (z|x) is an approximate distribution of posterior p(z|x) and φ is the model parameter of q.

We designate q φ (z|x) as encoder and p θ (x|z) as decoder. Moreover, in Equation 1, the first term represents a regularization. The second one represents a negative reconstruction error.

To optimize the lower bound L(x) with respect to parameters θ, φ, we estimate gradients of Equation 1 using stochastic gradient variational Bayes (SGVB). If we consider q φ (z|x) as Gaussian distribution N (z; µ, diag(σ2 )), where φ = {µ, σ 2 }, then we can reparameterize z ∼ q φ (z|x) to z = µ + σ ⊙ ǫ, where ǫ ∼ N (0, I). Therefore, we can estimate the gradients of the negative reconstruction term in Equation 1 with respect to θ and φ as

∇ θ,φ E q φ (z|x) [log p θ (x|z)] = E N (ǫ;0,I) [∇ θ,φ log p θ (z|µ + σ ⊙ ǫ)].

Because the gradients of the regularization term are solvable analytically, we can optimize Equation 1 with standard stochastic optimization methods.

3.2. JOINT MULTIMODAL VARIATIONAL AUTOENCODERS:

Next, we consider i.i.d. dataset (X, W) = {(x 1 , w 1 ), ..., (x N , w N )}, where two modalities x and w have different kinds of dimensions and structures 2 . Our objective is to generate two modalities bi-directionally. For that reason, we assume that these are conditioned independently on the same latent concept z: joint representation. Therefore, we assume their generating processes as z ∼ p(z) and x, w ∼ p(x, w|z) = p θx (x|z)p θw (w|z), where θ x and θ w represent the model parameters of each independent p. Figure 2 (a) shows a graphical model that represents generative processes. One can see that this models joint distribution of all modalities, p(x, w). Therefore, we designate this model as a joint multimodal variational autoencoder (JMVAE). Two approaches to estimate encoders with a single input, q(z|x) and q(z|w), on the JMVAE: left, make modalities except an input modality missing (JMVAE-zero); right, prepare encoders that have a single input and make them close to the JMVAE encoder (JMVAE-kl).

!" !# $ !" !# !$ !" !$ !" !$ !# !$ %& %& JMVAE-zero JMVAE-kl q(z|x) ≃ qφ(z|x, 0); q(z|w) ≃ qφ(z|0, w). (qφ(z|x, w) ||qφ x (z|x)) + ||qφ w (z|w))] !# x z w JMVAE-zero JMVAE-kl

Considering an approximate posterior distribution as q φ (z|x, w), we can estimate a lower bound of the log-likelihood log p(x, w) as follows:

EQUATION

Equation 3 has two negative reconstruction terms which are correspondent to each modality. As with VAEs, we designate q φ (z|x, w) as the encoder and both p θx (x|z) and p θw (w|z) as decoders.

We can apply the SGVB to Equation 3 just as Equation 1, so that we can parameterize the encoder and decoder as deterministic deep neural networks and optimize them with respect to their parameters, θ x , θ w , and φ. Because each modality has different feature representation, we should set different networks for each decoder, p θx (x|z) and p θw (w|z). The type of distribution and corresponding network architecture depends on the representation of each modality, e.g., Gaussian when the representation of modality is continuous, and a Bernoulli when it is a binary value.

Unlike original VAEs and CVAEs, the JMVAE models joint distribution of all modalities. In this model, modalities are conditioned independently on a joint latent variable. Therefore, we can extract better representation that includes all information of modalities. Moreover, we can estimate both marginal distribution and conditional distribution in bi-directional, so that we can not only obtain images reconstructed themselves but also draw texts from corresponding images and vice versa. Additionally, we can extend JMVAEs to handle more than two modalities such as p(x, w 1 , w 2 , ...) in the same learning framework.

3.3. INFERENCE MISSING MODALITIES:

In the JMVAE, we can extract joint latent features by sampling from the encoder q φ (z|x, w) at testing time. Our objective is to exchange modalities bi-directionally, e.g., images to texts and vice versa. In this setting, modalities that we want to sample are missing, so that inputs of such modalities are set to zero (the left panel of Figure 2(b) ). The same is true of reconstructing a modality only from itself. This is a natural way in discriminative multimodal settings to estimate samples from unimodal information (Ngiam et al., 2011) . However, if missing modalities are high-dimensional and complicated such as natural images, then the inferred latent variable becomes incomplete and generated samples might collapse.

We propose a method to solve this issue, which we designate as JMVAE-kl. Moreover, we describe the former way as JMVAE-zero to distinguish it. Suppose that we have encoders with a single input, q φx (z|x) and q φw (z|w), where φ x and φ w are parameters. We would like to train them by bringing their encoders close to an encoder q φ (z|x, w) (the right panel of Figure 2(b) ). Therefore, the object function of JMVAE-kl becomes

EQUATION

)

where α is a factor that regulates the KL divergence terms.

From another viewpoint, maximizing Equation 4 can be regarded as minimizing the variation of information with variational learning on parameterized distributions (proven and derived in Appendix A). The variation of information, a measure of the distance between two variables, is written as

-E pD (x,w) [log p(x|w) + log p(w|x)],

where p D is the data distribution. It is apparent that the variation of information is the sum of two negative conditional log-likelihoods. Therefore, minimizing the variation of information contributes to appropriate bi-directional exchange of modalities. Sohn et al. (2014) also train their model to minimize the VI for the same objective as ours. However, they use DBMs with MCMC training.

4. EXPERIMENTS:

This section presents evaluation of the qualitative and quantitative performance and confirms the JMVAE functionality in practice.

4.1. DATASETS:

As described herein, we used two datasets: MNIST and CelebA (Liu et al., 2015) .

MNIST is not a dataset for multimodal setting. In this work, we used this dataset for toy problem of multimodal learning. We consider handwriting images and corresponding digit labels as two different modalities. We used 50,000 as training set and the remaining 10,000 as a test set.

CelebA consists of 202,599 color face images and corresponding 40 binary attributes such as male, eyeglasses, and mustache. In this work, we regard them as two modalities. This dataset is challenging because these have completely different kinds of dimensions and structures. Beforehand, we cropped the images to squares and resized to 64 × 64 and normalized. From the dataset, we chose 191,899 images that are identifiable face by OpenCV and used them for our experiment. We used 90% out of all the dataset contains as training set and the remaining 10% of them as test set.

4.2. MODEL ARCHITECTURES:

For MNIST, we considered images as x ∈ R 28×28 and corresponding labels as w ∈ {0, 1} 10 . We prepared two networks each with two dense layers of 512 hidden units and using leaky rectifiers and shared the top of each layers and mapped them into 64 hidden units. Moreover, we prepared two networks each with three dense layers of 512 units and set p(x|z) as Bernoulli and p(w|z) as categorical distribution whose output layer is softmax. We used warm-up (Bowman et al For CelebA, we considered face images as x ∈ R 32×32×3 and corresponding attributes as w ∈ {-1, 1} 40 . We prepared two networks with layers (four convolutional and a flattened layers for x and two dense layers for w) with ReLU and shared the top of each layers and mapped them into 128 units. For the decoder, we prepared two networks, with a dense and four deconvolutional layers for x and three dense layers for w, and set Gaussian distribution for decoder of both modalities, where the variance of Gaussian was fixed to 1 for the decoder of w. In CelebA settings, we combined JMVAE with generative adversarial networks (GANs) (Goodfellow et al., 2014) to generate clearer images. We considered the network of p(x|z) as generator in GAN, then we optimized the GAN loss with the lower bound of the JMVAE, which is the same way as a VAE-GAN model (Larsen et al., 2015) . As presented herein, we describe this model as JMVAE-GAN. We set N t = 20 and trained for 100 epochs on CelebA.

We used the Adam optimization algorithm (Kingma & Ba, 2014) with a learning rate of 10 -3 on MNIST and 10 -4 on CelebA. The models were implemented using Theano (Team et al., 2016) and Lasagne (Dieleman et al., 2015) . For this experiment, we estimated test log-likelihood to evaluate the performance of model. This estimate roughly corresponds to negative reconstruction error. Therefore, higher is better. From this performance, we can find that not only whether the JMVAE can generate samples properly but also whether it can obtain joint representation properly. If the log-likelihood of a modality is low, representation for this modality might be hurt by other modalities. By contrast, if it is the same or higher than model trained on a single modality, then other modalities contribute to obtaining appropriate representation.

We estimate the test marginal log-likelihood and test conditional log-likelihood on JMVAE. We compare the test marginal log-likelihood against VAEs (Kingma & Welling, 2013; Rezende et al., 2014) and the test conditional log-likelihood against CVAEs (Kingma et al., 2014; Sohn et al., 2015) and CMMAs (Pandey & Dukkipati, 2016) . On CelebA, we combine all competitive models with GAN and describe them as VAE-GAN, CVAE-GAN, and CMMA-GAN. For fairness, architectures and parameters of these competitive models were set to be as close as possible to those of JMVAE.

We calculate the importance weighted estimator (Burda et al., 2015) from lower bounds at testing time because we would like to estimate the true test log-likelihood from lower bounds. To estimate the test marginal log-likelihood p(w) of the JMVAE, we use two possible lower bounds: sampling from q φ (z|x, w) or q φx (z|x). We describe the former lower bound as the multiple lower bound and the latter one as the single lower bound. When we estimate the test conditional loglikelihood log p(x|w), we also use two lower bounds, each of which is estimated by sampling from q φ (z|x, w) (multiple) or q φw (z|w) (single) (see Appendix B for more details). To estimate the single lower bound, we should approximate the single encoder (q φx (z|x) or q φw (z|w)) by JMVAE-zero or JMVAE-kl. When the value of log-likelihood with the single lower bound is the same or larger than that with the multiple lower bound, the approximation of the single encoder is good. Note that original VAEs use a single lower bound and that CVAEs and CMMAs use a multiple lower bound.

4.3.2. MNIST:

Our first experiment evaluated the test marginal log-likelihood and compared it with that of the VAE on MNIST dataset. We trained the model with both JMVAE-zero and JMVAE-kl and confirmed these differences. As described in Section 4.3.1, we have two possible ways of estimating the marginal log-likelihood of the JMVAE, i.e., multiple and single lower bounds. The left of Table 1 shows the test marginal log-likelihoods of the VAE and JMVAE. It is apparent that log-likelihood of the JMVAE-zero is the same or slightly better than that of the VAE. In the case of the loglikelihood of JMVAE-kl, the log-likelihood becomes better as α is small. Especially, JMVAE-kl with α = 0.01 and single lower bound archives the highest log-likelihood in Table 1 . If α is 1, however, then the test log-likelihood on JMVAE-kl becomes much lower. This is because the influence of the regularization term becomes strong as α is large. Points with different colors correspond to the digit labels. These were sampled from q(z|x) in the VAE and q(z|x, w) in both the CVAE and JMVAE.

We used JMVAE-zero as the JMVAE.

Next, we evaluated the test conditional log-likelihood and compared it with that of the CVAE and CMMA conditioned on w. As in the case of the marginal log-likelihood, we can estimate the JMVAE's conditional log-likelihood by both the single and multiple lower bound. The single bound can be estimated using JMVAE-zero or JMVAE-kl. The right of Table 1 shows the test conditional log-likelihoods of the JMVAE, CVAE, and CMMA. It is apparent that the CVAE achieves the highest log-likelihood. Even so, in the case of multiple bound, log-likelihoods with both JMVAE-zero and JMVAE-kl (except α = 1) outperform that of the CMMA.

It should be noted that the log-likelihood with JMVAE-zero and single bound is significantly low. As described in Section 3.3, this is because a modality w is missing as input. By contrast, it is apparent that the log-likelihood with JMVAE-kl is improved significantly from that with JMVAEzero. It shows that JMVAE-kl solves the issue of missing modalities (we can also find this result in generated images, see Appendix E). Moreover, we find that this log-likelihood becomes better as α is large, which is opposite to the other results. Therefore, there is a trade-off between whether each modality can be reconstructed properly and whether multiple modalities can be exchanged properly and it can be regulated by α.

4.3.3. CELEBA:

In this section, we used CelebA dataset to evaluate the JMVAE. 1 shows that the improvement on CelebA is greater than that on MNIST, which suggests that joint representation with multiple modalities contributes to improvement of the quality of the reconstruction and generation in the case in which an input modality is large-dimensioned and complicated.

4.4. QUALITATIVE EVALUATION:



4.4.1. JOINT REPRESENTATION ON MNIST:

In this section, we first evaluated that the JMVAE can obtain joint representation that includes the information of modalities. Figure 3 shows the visualization of latent representation with the VAE, CVAE, and JMVAE on MNIST. It is apparent that the JMVAE obtains more discriminable latent representation by adding digit label information. Figure 3 (b) shows that, in spite of using multimodal information as with the JMVAE, points in CVAE are distributed irrespective of labels because CVAEs force latent representation to be independent of label information, i.e., it is not objective for CVAEs to obtain joint representation.

4.4.2. GENERATING FACES FROM ATTRIBUTES AND JOINT REPRESENTATION ON CELEBA:

Next, we confirm that JMVAE-GAN on CelebA can generate images from attributes. Figure 4 (a) portrays generated faces conditioned on various attributes. We find that we can generate an average face of each attribute and various random faces conditioned on a certain attributes. Figure 4(b) shows that samples are gathered for each attribute and that locations of each variation are the same irrespective of attributes. From these results, we find that manifold learning of joint representation with images and attributes works well. 

4.4.3. BI-DIRECTIONAL GENERATION BETWEEN FACES AND ATTRIBUTES ON CELEBA:

Finally, we demonstrate that JMVAE-GAN can generate bi-directionally between faces and attributes. Figure 5 shows that MVAE-GAN can generate both attributes and changed images conditioned on various attributes from images which had no attribute information. This way of generating an image by varying attributes is similar to the way of the CMMA (Pandey & Dukkipati, 2016) . However, the CMMA cannot generate attributes from an image because it only generates images from attributes in one direction.

5. CONCLUSION AND FUTURE WORK:

In this paper, we introduced a novel multimodal learning model with VAEs, the joint multimodal variational autoencoders (JMVAE). In this model, modalities are conditioned independently on joint representation, i.e., it models a joint distribution of all modalities. We further proposed the method (JMVAE-kl) of reducing the divergence between JMVAE's encoder and a prepared encoder of each modality to prevent generated samples from collapsing when modalities are missing. We confirmed that the JMVAE can obtain appropriate joint representations and high log-likelihoods on MNIST and CelebA datasets. Moreover, we demonstrated that the JMVAE can generate multiple modalities bi-directionally on the CelebA dataset.

In future work, we would like to evaluate the multimodal learning performance of JMVAEs using various multimodal datasets such as containing three or more modalities. We can also estimate test conditional log-likelihood p(x|w) from these two lower bounds as L single (x|w) = E q φw (z|w) [log p(x, z|w) q φw (z|w) ] = E q φw (z|w) [log p θx (x|z)p θw (w|z)p(z) q φw (z|w) ]log p(w), (9)

L multiple (x|w) = E q φ (z|x,w) [log p θx (x|z)p θw (w|z)p(z) q φ (z|x, w) ]log p(w),

where log p(w) = log E p(z) [p θw (w|z)] = log 1 Nw Nw i p θw (w|z (i) ) and z (i) ∼ p(z). In this paper, we set N w = 5, 000 on MNIST and N w = 10 on CelebA.

We can obtain a tighter bound on the log-likelihood by k-fold importance weighted sampling. For example, we obtain an importance weighted bound on log p(x) from Equation 11 

Strictly speaking, these two lower bounds are not equal. However, if the number of importance samples is extremely large, the difference of these two lower bounds converges to 0.

Proof. Let the multiple and single k-hold importance weighted lower bounds as L k single and L k single . From the theorem of the importance weighted bound, both L k single and L k multiple converge to log p(x) as k → ∞. 

:

According toLouizos et al. (2015), this independence might not be satisfied strictly because the encoder in CVAEs still has the dependence.

In our experiment, these depend on dataset, see Section 4.2.

https://en.wikipedia.org/wiki/Mona Lisa 4 https://en.wikipedia.org/wiki/Wolfgang Amadeus Mozart

Figures and Tables:

Figure 2: (a) Graphical model of the JMVAE. Gray circles represent observed variables. The white one denotes a latent variable. (b) Two approaches to estimate encoders with a single input, q(z|x) and q(z|w), on the JMVAE: left, make modalities except an input modality missing (JMVAE-zero); right, prepare encoders that have a single input and make them close to the JMVAE encoder (JMVAE-kl).

Figure 3: Visualizations of 2-D latent representation. The network architectures are the same as those in Section 4.3, except that the dimension of the top hidden layer is forced into 2.Points with different colors correspond to the digit labels. These were sampled from q(z|x) in the VAE and q(z|x, w) in both the CVAE and JMVAE. We used JMVAE-zero as the JMVAE.

Figure 4: (a) Generation of average faces and corresponding random faces. We first set all values of attributes {-1, 1} randomly and designate them as Base. Then, we choose an attribute that we want to set (e.g., Male, Bald, Smiling) and change this value in Base to 2 (or -2 if we want to set "Not"). Each column corresponds to same attribute according to legend. Average faces are generated from p(x|zmean), where zmean is a mean of q(z|w). Moreover, we can obtain various images conditioned on the same values of attributes such as x ∼ p(x|z), where z = zmean + σ ⊙ ǫ, ǫ ∼ N (0, ζ), and ζ is the parameter which determines the range of variance. In this figure, we set ζ = 0.6. Each row in random faces has the same ǫ. (b) PCA visualizations of latent representation. Colors indicate which attribute each sample is conditioned on.

Figure 6: Comparison of the original images and reconstructed images by the JMVAE (α = 0.1). We used (a) MNIST and (b) CelebA datasets.

as follows:log p(x) ≥ E z1,...,z k ∼q φx (z|x) [log 1 k k i=1 p θx (x|z)p(z) q φx (z|x) ] = L k single (x).

Figure 6 presents a comparison of the original image and reconstructed image by the JMVAE on both MNIST and CelebA datasets. It is apparent that the JMVAE can reconstruct the original image properly with either a multiple or single encoder.

We show qualitatively and quantitatively that JMVAE can extract appropriate joint distribution and that it can generate and reconstruct modalities similarly or more properly than conventional VAEs.• We demonstrate that the JMVAE can generate multiple modalities bi-directionally even if these modalities have completely different kinds of dimensions and structures, e.g., highdimentional color face images and low-dimentional binary attributes.

Table content: <table/>

., 2015; Sønderby et al., 2016), which first forces training only of the term of the negative reconstruction error and then gradually increases the effect of the regularization term to prevent local minima during early training. We increased this term linearly during the first N t epochs as with Sønderby et al. (2016). We set N t = 200 and trained for 500 epochs on MNIST. Moreover, same as Burda et al. (2015); Sønderby et al. (2016), we resampled the binarized training values randomly from MNIST for each epoch to prevent over-fitting.

Table content: <table/>

Evaluation of test log-likelihood. All models are trained and tested on MNIST. α is a coefficient of regularization term in JMVAE-kl (Equation4): left, marginal log-likelihood; right, conditional log-likelihood.

Table content: <table><tr><td>VAE JMVAE-zero JMVAE-kl, α = 0.01 JMVAE-kl, α = 0.1 JMVAE-kl, α = 1</td><td>≤ log p(x) multiple single -86.91 -86.89 -86.89 -86.89 -86.55 -86.86 -86.73 -89.20 -89.20</td><td>CVAE CMMA JMVAE-zero JMVAE-kl, α = 0.01 JMVAE-kl, α = 0.1 JMVAE-kl, α = 1</td><td>≤ log p(x|w) multiple single -83.80 -86.12 -84.64 -4838 -84.61 -129.6 -84.72 -126.0 -86.97 -112.7</td></tr></table>

Evaluation of log-likelihood. Models are trained and tested on CelebA. We trained JMVAE-kl and set α = 0.1: left, marginal log-likelihood; right, conditional log-likelihood (with the multiple lower bound).

Table content: <table><tr><td/><td colspan="2">≤ log p(x)</td><td/><td>≤ log p(x|w)</td></tr><tr><td/><td colspan="2">multiple single</td><td>CVAE-GAN</td><td>-4152</td></tr><tr><td>VAE-GAN</td><td/><td>-4439</td><td>CMMA-GAN</td><td>-4147</td></tr><tr><td>JMVAE-GAN</td><td>-4141</td><td>-4144</td><td>JMVAE-GAN</td><td>-4130</td></tr><tr><td colspan="2">4.3 QUANTITATIVE EVALUATION</td><td/><td/><td/></tr><tr><td>4.3.1 EVALUATION METHOD</td><td/><td/><td/><td/></tr></table>

Table 2 presents the evaluations of marginal and conditional log-likelihood. From this table, it is apparent that values of both marginal and conditional log-likelihood with JMVAEs are larger than those with other competitive methods. Moreover, comparison with Table

Table content: <table/>

Peer review (iclr2017):
The paper introduces the joint multimodal variational autoencoder, a directed graphical model for modeling multimodal data with latent variable. the model is rather straightforward extension of standard VAE where two data modalities are generated from a shared latent representation independently. In order to deal with missing input modalities or bi-directional inference between two modalities the paper introduces modality-specific encoder that is trained to minimize the KL divergence of latent variable distributions between joint and modality-specific recognition networks. The paper demonstrates its effectiveness on MNIST and CelebA datasets, both in terms of test log-likelihoods and the conditional image generation and editing. The proposed method is rather straightforward extension of VAE and therefore the model should inherent the probabilistic inference methods of VAE. For example, for missing data modalities, the model should be able to infer joint representation as well as filling in the missing modalities via iterative sampling as introduced by Rezende et al. (2014). Given marginal improvement, I am not convinced by the contribution of modality-specific encoders in Section 3.3. In addition, the inference methods introduced for generating Figure 5 looks somewhat unprincipled; I am wondering the conditional image generation results by following more principled approach (e.g., iterative sampling). Experimental results on joint image-attribute generation is also missing.