Year: 2021

Abstract:
To obtain a joint representation from multimodal data in variational autoencoders (VAEs), it is important to infer the representation from arbitrary subsets of modalities after learning. A scalable way to achieve this is to aggregate the inferences of each modality as experts. A state-of-the-art approach to learning this aggregation of experts is to encourage all modalities to be reconstructed and crossgenerated from arbitrary subsets. However, this learning may be insufficient if cross-generation is difficult. Furthermore, to evaluate its objective function, exponential generation paths concerning the number of modalities are required. To alleviate these problems, we propose to explicitly minimize the divergence between inferences from arbitrary subsets and the surrogate joint posterior that approximates the true joint posterior. We also proposed using a gradient origin network, a deep generative model that learns inferences without using an inference network, thereby reducing the need for additional parameters by introducing the surrogate posterior. We demonstrate that our method performs better than existing scalable multimodal VAEs in inference and generation.

heading: 1 INTRODUCTION
text: Our world is characterized by many kinds of information. This multimodal information can be used to form common concepts. For instance, by recognizing information, such as the landscape of the beach and the salty scent of the ocean, we can form a concept about the sea. In multimodal learning (Baltrušaitis et al., 2018), it has been studied as the learning of joint representations from multimodal data, especially using variational autoencoders (VAEs) (Kingma & Welling, 2013; Rezende et al., 2014).
The goal of multimodal representation learning with VAEs (Suzuki et al., 2016; Vedantam et al., 2018; Wu & Goodman, 2018; Shi et al., 2019) is to learn all given multimodal data and to infer a joint representation from arbitrary subsets of modalities. This representation is useful as an input to predict the corresponding label information or as an intermediate representation to generate other modalities, i.e., cross-generations. However, this is difficult because a joint representation must be inferred even when some of the modalities are missing at test time. The most naive way to deal with this difficulty is to have inference models for all possible combinations of modalities, which is exponentially necessary given the number of modalities.
A scalable way to address this is to consider the inferences for each modality as experts and combine them to approximate the inferences from an arbitrary subset of modalities. There are two types of methods for aggregating inferences: products of experts (PoE) (Hinton, 2002) and mixtures of experts (MoE); VAEs that use these as approximate posteriors are proposed as MVAE (Wu & Goodman, 2018) and MMVAE (Shi et al., 2019), respectively. Each method has different advantages that exist in a trade-off relationship, i.e., PoE increases the amount of mutual information with joint representation by increasing the number of modalities, while MoE has the effect of encouraging cross-generation between modalities.
Sutter et al. (2021) defined the objective of multimodal VAEs to be that the inference of all possible subsets approximate the true joint posterior, and derived a generalized evidence lower bound (ELBO) as a lower bound of that objective. This ELBO can be viewed as using a mixture of products of experts (MoPoE) as an aggregation method for inference, which is to generate all modalities from PoE-aggregated inference of arbitrary subsets. In other words, this model, called MoPoE-VAE,
tries to achieve the above objective of multimodal VAEs by minimizing all reconstruction and crossgeneration errors in all modality combinations. They showed that this is a generalization that has the advantages of both PoE and MoE. However, these properties of MoPoE-VAE imply that inference may not be sufficiently learned when cross-generation is difficult. Furthermore, the number of paths for this reconstruction and cross-generation in MoPoE-VAE is required to be exponential with respect to the number of modalities. Therefore, we pursue a scalable method that realizes the above objective of multimodal VAEs.
In this paper, we propose a different approach to achieve this objective. We first prepare a surrogate joint posterior, which approximates the true joint posterior given all modalities, and then minimize the divergence between the PoE-aggregated inferences and this surrogate posterior. The approximation of surrogate posterior is perform by the usual VAE optimization, and we simultaneously perform this optimization and the divergence minimization above.
The advantages of this new approach are twofold. First, if the surrogate model is sufficiently close to the true posterior, then optimizing this objective function explicitly achieves the objective of multimodal VAEs defined in MoPoE, i.e, approximating inferences from arbitrary subsets to the true posterior. Second, unlike MoPoE, this method is not exponentially more expensive. This is because this method is based on divergence minimization of all inference models, rather than reconstruction or cross-generation of all modalities in combination. Although this method requires an additional surrogate model, the size is not exponential as in previous similar methods (Suzuki et al., 2016; Vedantam et al., 2018), but only doubles the number of parameters in encoders. Therefore, we call this method a scalable multimodal VAE (SMVAE).
Furthermore, to reduce the number of parameters in the surrogate posterior, we propose to use variational gradient origin networks (VGONs) (Bond-Taylor & Willcocks, 2020) for training the surrogate posterior. VGONs are deep generative models that can rapidly acquire amortized inference to representations without a trainable network. We call our method that uses VGONs for surrogate posterior SMVGON.
Experiments show that the proposed method performs better than conventional methods in inferring joint representations and generating modalities on commonly used multimodal benchmarks. In particular, we show that the proposed method improves the performance in situations where the conventional methods have difficulty in cross-generation. We also confirm that the performance of SMVGONs with VGONs is comparable to that of SMVAEs when the input modalities are relatively simple, without the need for additional inference networks for surrogate posteriors.

heading: 2 PRELIMINARY

heading: 2.1 MULTIMODAL VAES
text: Suppose that we are given an i.i.d. dataset {X(i)}Ni=1, where each example is a set of M modalities X(i) = {x (i) m }Mm=1 and where each modality x (i) m = [x (i) 1m, ..., x (i) Dmm
] has its own dimension Dm and domain. We assume that these examples are derived from data distribution pdata(X) and that each example X(i) has a corresponding common latent concept z(i) = [z (i) 1 , ..., z (i) j , ..., z (i) J ] ∈ R
J , i.e., a joint representation.
Given a training set {X(i)}Ni=1, our objective is to infer a joint representation from the subset of modalities1 Xs ⊆ X and to generate another subset X ′ s ⊆ X via that representation. In other words, we aim to obtain an inference p(z|X) and a joint distribution p(X, z) = ∏
xm∈X p(xm|z)p(z) using
a training set. In this study, we designate the generation when Xs = X ′ s as reconstruction and when Xs ∩X ′ s = ∅ as cross-generation.
To achieve this, we use VAEs (Kingma & Welling, 2013; Rezende et al., 2014), which are deep generative models that can learn inference in addition to generative models. Therefore, we can acquire representations of data by inference. The generative model pΘ(X|z) = ∏
xm∈X pθm(xm|z), where
Θ = {θm} M m=1 is a set of parameters, is parameterized by deep neural networks, and the prior of
1We denote the multimodal set and any subset of it by X and Xs, respectively. We also omit the index of each example for brevity.
the latent variable is set as a standard Gaussian p(z) = N (0, I). The objective function of VAEs is to maximize the marginal log-likelihood of the multimodal dataset: ∑N
i=1 log pΘ(Xi). However, since it is tractable to optimize this likelihood directly, we introduce an inference model qΦ(z|X) to approximate the posterior pΘ(z|X), and instead optimize the following lower bound of the marginal log-likelihood given X , which is called the ELBO:
L(Θ,Φ;X) ≡EqΦ(z|X)[log pΘ(X|z)]−DKL(qΦ(z|X)||p(z)). (1)
Here, the optimization of ELBO with respect to the parameters of the inference model corresponds to minimizing the following Kullback-Leiber (KL) divergence:
DKL(qΦ(z|X)||pΘ(z|X)). (2)
In other words, maximizing the ELBO encourages bringing the true posterior closer to the inference model.
After optimizing Equation 1, we aim to infer a joint representation z from an arbitrary subset of modalities Xs. However, this is challenging because Equation 1 only models joint inferences from all modalities. Therefore, for inferences from a subset of modalities Xs, other modality inputs are missing, which results in the inability to infer the representation properly (Suzuki et al., 2016).
A scalable way to alleviate this difficulty is to consider the inference model for each modality as an expert and approximate the inference model for arbitrary modalities by combining them. There are two ways to combine experts: product and mixture.

heading: 2.1.1 AGGREGATED INFERENCE WITH POE AND MOE
text: A joint inference given all modalities X is expressed as the following using PoE (Hinton, 2002):
qPoE(z|X) ∝ p(z) ∏
xm∈X
qφm(z|xm). (3)
In general, PoE cannot be calculated as closed-form, but if we restrict each expert to Gaussian qφm(z|xm) = N ( µm, diag(σ 2 m)) ) , then Equation 3 is calculable as qPoE(z|X) = N (µ, diag(σ2)), where µj = ∑ m µjmσ −2
jm∑ m σ −2 jm and σ2j = 1∑ m σ −2 jm . Therefore, the Gaussian PoE
enables us to infer the representation from subset Xs.
However, if we optimize this inference model with Equation 1, then the inference of arbitrary modalities is not optimized explicitly. Wu & Goodman (2018) proposed to train ELBOs with PoEaggregated inferences of all modality combinations ∑
Xs∈X L(Θs,Φs;Xs); however, since the
number of ELBOs increases exponentially, they used the subsampled version. The model that learns PoE inference by this objective function is called MVAE (Wu & Goodman, 2018).
Another method to combine experts is to use MoE:
qMoE(z|X) = ∑
xm∈X
wmqφm(z|xm), (4)
where wm is a mixing coefficient of each expert and where ∑
xm∈X wm = 1. Shi et al. (2019)
proposed a VAE with MoE-aggregated inference models called MMVAE.
There is a trade-off between the approximate posterior with PoE and MoE. PoE can perform inference given any subset and MVAE can learn inference explicitly during training. PoE also has the property that increasing the size of the subset, i.e., the number of input modalities, leads to an increase in the information content of the inferred latent variables. However, the objective function of MVAE does not guarantee that the inferences from any subset are the same, since it only learns reconstruction and not cross-generation. Sutter et al. (2021) also pointed out that its objective function by subsampling is not a valid lower bound.
MMVAE based on MoE, on the other hand, has a valid ELBO and learns to explicitly encourage cross-generation, which ensures that the inferences from each modality remain consistent. However, this method only learns inferences from a single modality, and increasing the number of modalities does not necessarily increase the information content of the representation.

heading: 2.1.2 MOPOE-VAE
text: Sutter et al. (2021) considered optimization on all possible subsets, i.e., the powerset P(X), and reestablished the minimization of the following KL divergence as the objective of multimodal VAEs:
argmin φ
∑
Xc∈P(X)
DKL (qφc (z|Xk) ‖pΘ(z|X)) . (5)
They then proposed a generalized ELBO that is a valid lower bound on the log-likelihood pΘ(X) and whose maximization corresponds to the minimization of Equation 5:
EqMoPoE(z|X)[log pΘ(X|z)]−DKL(qMoPoE(z|X)||p(z)), (6)
where the form of the posterior of qMoPoE(z|X) is called MoPoE and is defined as follows:
qMoPoE(z|X) = 1
2M
∑
Xc∈P(X)
qPoE(z|Xc). (7)
A feature of the model that maximizes Equation 6, called MoPoE-VAE, is that it learns to generate all modalities from all possible subsets, including reconstruction and cross-generation, which improves the performance of inference and generation from all subsets. In addition, they showed that this MoPoE-VAE is a generalization of MVAE and MMVAE, specifically MVAE when the summation in Equation 7 is taken only for a set of all modalities X and MMVAE when it is taken for the single modalities {xm} M m=1. Therefore, MoPoE-VAE can be regarded as having the advantages of both MVAE and MMVAE.

heading: 2.2 VARIATIONAL GRADIENT ORIGIN NETWORKS
text: VGONs (Bond-Taylor & Willcocks, 2020) are deep generative models that can perform amortized inference equivalent to VAEs without an inference network.2
Given an observed variable x and a latent variable z, let the generative model be pθ(x|z), and the inference model be qφ,z0(z) = N (µφ(z0), σ 2 φ(z0)I) with the mean and variance mapped by trainable linear transformations (µφ and σ 2 φ) from a parameter z0. This objective function, ELBO, given x is defined as follows:
LV GON (θ, φ, z0;x) ≡Eqφ,z0 (z)[log pθ(x|z)]−DKL(qφ,z0(z)||p(z)). (8)
In VGONs, the estimator for z0 is calculated by a single update based on the gradient of Equation 8:
ẑ0 = z0 +∇z0LV GON (θ, φ, z0;x), (9)
where z0 = 0, i.e, the initial value of z0 is set to the origin. Using this estimator ẑ0, we can obtain inference as z ∼ qφ,ẑ0(z) and can evaluate the objective function as LV GON (θ, φ, ẑ0;x). In learning, we also backpropagate the gradient of the objective function through Equation 9, which allowed us to learn both inference and generation simultaneously as in VAEs.

heading: 3 METHODS

heading: 3.1 ISSUES OF MOPOE-VAE
text: MoPoE-VAE aims at the objective of Equation 5 by maximizing the ELBO of Equation 6. The two are equivalent in variational inference, but have different implications for learning including generative models, i.e., whether the Equation 5 is achieved depends on the performance of all modalities of reconstruction and cross-generation. Therefore, in situations where cross-generation is difficult, this objective might not be fully achieved, specifically, the inference and generation of some modalities might be significantly worse than others.
2In the original paper (Bond-Taylor & Willcocks, 2020), gradient origin networks were introduced first, and then variational GONs and implicit GONs were introduced as an extension of them.
Another issue is that the computational cost of evaluating Equation 6 increases exponentially with the number of modalities. Specifically, we consider a computational path that generates modalities from inferred representations. In this case, since there are 2M representations to be inferred by MoPoE, the generation path required to evaluate Equation 6 is M × 2M ; therefore, as the number of modalities increases, the evaluation of the ELBO becomes more intractable and might also lead to insufficient learning of cross-generation.

heading: 3.2 A SURROGATE JOINT POSTERIOR AND THE REFORMULATION OF THE OBJECTIVE
text: To address the above issues, we aim to explicitly realize Equation 5. However, since a true posterior cannot be obtained, we prepare a surrogate joint posterior qΛ(z|X), which is a Gaussian distribution parameterized by a neural network and should approximate the true posterior pΘ(z|X). We then reformulate our objective as minimizing the KL divergence between subset inferences and this surrogate posterior:
argmin φ
∑
Xc∈P(X)
DKL (qφc (z|Xc) ‖qΛ(z|X)) . (10)
Here, we consider each inference from a subset qφc (z|Xc) to be expressed in PoE form. Then, since the surrogate posterior is defined as Gaussian, this KL divergence can be calculated analytically.
If DKL(qΛ(z|X)||pΘ(z|X)) ≈ 0 is satisfied, then this optimization problem is the same as Equation 5. As explained in Section 2.1, minimizing the KL divergence between the approximate posterior (in this case, the surrogate posterior) and the true posterior is equivalent to maximizing the ELBO. Therefore, we can first maximize the ELBO L(Θ,Λ;X) sufficiently, and then fix the surrogate posterior and perform the optimization of Equation 10. However, this requires a two-step learning process, which can be time consuming. Therefore, we propose to optimize these two steps simultaneously, referring to existing multimodal VAEs with additional inference models such as JMVAE (Suzuki et al., 2016) (see Section 4), i.e., maximize the following objective function:
L(Θ,Λ;X)− β
2M
∑
Xc∈P(X)
DKL (qφc (z|Xc) ‖qΛ(z|X)) , (11)
where β is a hyperparameter that adjusts for the effect of KL divergence in the objective function. We train this objective function in an end-to-end fashion. Empirically, we have found that this end-toend optimization results in higher performance than the two-stage optimization (see Appendix D). At test time, we infer representations from subsets using a PoE-aggregated inference model qφc (z|Xc).
The advantages of this objective function are as follows. First, inference models from arbitrary subsets are learned not by cross-generation, but by explicitly minimizing the divergence between them and the surrogate posterior. Moreover, the surrogate posterior is learned based on the maximization of ELBO L(Θ,Λ;X), i.e., based on the reconstruction in all modalities, rather than crossgeneration. Therefore, this objective can mitigate the impact of the difficulty of cross-generation on the performance of inference and generation.
Secondly, the computational cost can be significantly reduced compared to MoPoE-VAE. There is only one pass for generation in evaluating this objective function for each modality in L(Θ,Λ;X). In addition, the second term of Equation 11 computes the inference for all possible subsets, which is obtained by aggregating the inferences for each modality; therefore, the inference pass is also only once for each modality.
We call the proposed model a scalable multimodal VAE (SMVAE) because it is scalable to the number of modalities while explicitly optimizing toward the objective of multimodal VAEs.

heading: 3.3 REDUCING THE PARAMETERS IN SURROGATE INFERENCE MODELS
text: Although the proposed method alleviate the issues of previous multimodal VAEs, it also has the disadvantage of requiring more parameters for the inference model. The added surrogate posterior requires as many parameters as all of the unimodal inference models, i.e., the parameter size of the inference model is twice as large as that of previous models such as MoPoE-VAE. Note that in existing studies of multimodal VAEs with additional inference models, such as JMVAE, the parameter
size increases exponentially because inference models are required for each subset. In comparison, SMVAE keeps the number of required parameters linear with respect to the number of modalities.
To reduce the number of required parameters, we propose to use VGONs to train the surrogate posterior and generative models for all modalities. In the VGON framework, the surrogate posterior given multimodal inputs can be defined as qλ,z0(z) = N (µλ(z0), σ 2 λ(z0)I), where the estimator of z0 is calculated as ẑ0 = ∇z0LV GON (Θ, λ, z0;X). Therefore, the objective function of the proposed method using VGONs is as follows:
LV GON (Θ, λ, z0;X)− β
2M
∑
Xc∈P(X)
DKL (qφc (z|Xc) ‖qλ,z0(z)) . (12)
We refer to this model as SMVGON to distinguish it from SMVAE.
Note that the surrogate posterior is used to approximate the aggregated inference model during training and is not used during testing; therefore, it is possible to use other iterative inference algorithms. In this study, however, we used VGONs, which can perform inference in a single iteration, to avoid slow training.

heading: 4 RELATED WORKS
text: For learning unsupervised joint representations from multimodal data, autoencoder-based methods have been used (Ngiam et al., 2011; Silberer & Lapata, 2014); however, they have difficulties in the challenge of missing modalities. Joint representation learning using deep belief networks (Hinton, 2009) or deep Boltzmann machines (Salakhutdinov & Hinton, 2009) can adequately handle missing modalities (Srivastava & Salakhutdinov, 2012; Srivastava et al., 2012; Sohn et al., 2014), but has difficulty scaling to large dataset.
Early multimodal learning using VAEs dealt with cross-generation of two modalities, such as conditional VAEs (Sohn et al., 2015; Kingma et al., 2014) and conditional multi-modal autoencoders (Pandey & Dukkipati, 2017). However, because these methods directly learn conditional generative models, they cannot perform bidirectional generation between modalities, nor can they obtain joint latent representations of different modalities.
Some studies of multimodal VAEs require additional inference models (Suzuki et al., 2016; Vedantam et al., 2018; Korthals et al., 2019; Wu & Goodman, 2019). Suzuki et al. (2016) pointed out that VAEs trained on all modalities have the problem of missing modalities in inference, and propose to approximate the unimodal inference models to explicitly approach the joint inference model of VAEs. The objective function of JMVAE is the ELBO of VAE given multimodal inputs minus the KL divergence corresponding to the above approximation.3 Vedantam et al. (2018) introduced TELBO, where the objective function is composed of the sum of ELBOs that take each set of modalities as input, using the same models as JMVAE. Korthals et al. (2019) introduced M2VAE that can be regarded as a combination of JMVAE and TELBO, and Wu & Goodman (2019) proposed VAEVAE that excludes a KL divergence term between joint inference and prior from it. They are learned end-to-end with a single objective function, except for TELBO.
The challenge with these models is that they require inference models for all modality combinations, i.e., the number of inference models grows exponentially with the number of modalities. Our proposed method is similar to these methods; however, unlike these methods, required memory cost is linear in the number of modalities. Kutuzova et al. (2021) proposed to introduce additional unimodal inference models instead of a joint inference model, and to reduce the number of required inference models to the square of the number of modalities. However, this also does not scale with the number of modalities compared to our linear cost model.
MMJSD proposed by Sutter et al. (2020) is closer to our work, i.e., it encourages unimodal inference to be closer to joint inference from all modalities. The differences between our work and
3When the number of modalities is two, our proposed method is similar to JMVAE, except that the direction of KL divergence is reversed and PoE-aggregation is used for inference. Please refer to Appendix A for the similarity and performance difference between JMVAE and SMVAE. When the number of modalities is two, the performance of the representations is comparable, but SMVAE outperforms in the cross-generation.
MMJSD are twofold. First, while MMJSD only considers unimodal inference, our work approximates joint inference with any combination of unimodals, which allows to learn inferences from arbitrary modalities. The second is the design of a joint inference. MMJSD considers a dynamic prior that combines all unimodal inferences and considers it a joint inference. However, there is no guarantee that this dynamic prior approximates the true posterior as in the surrogate posterior of SMVAE. In addition, since MMJSD uses the same networks for a joint inference and unimodal inferences, it might be difficult for this joint inference to achieve all of the approximations to the true posterior and to inferences from arbitrary modalities.
The other topic in multimodal VAEs is the acquisition of modality-specific latent variables, apart from the global latent variables that embed all modalities (Tsai et al., 2018; Hsu & Glass, 2018; Sutter et al., 2020). Our study is in a different direction from these, since we are addressing the possibility of inferring a single joint representation from arbitrary subset of modalities. In the experiments, we confirm that SMVAE achieves high performance with a single latent variable in a task where the conventional method achieves by adding modality-specific latent variables.
Recently, contrastive-based multimodal (multi-view) learning (Tian et al., 2019; Alayrac et al., 2020; Tsai et al., 2021) has been gaining attention instead of autoencoder-based, but these aim to acquire coordinated representations (Baltrušaitis et al., 2018), i.e., they do not assume inference from a subset of arbitrary modalities.

heading: 5 EXPERIMENTS
text: We use MNIST-SVHN-Text (Sutter et al., 2020), Bimodal CelebA (Sutter et al., 2020), and PolyMNIST (Sutter et al., 2021) datasets for evaluation. MNIST-SVHN-Text is the MNISTSVHN (Shi et al., 2019) with the addition of the text modality, which represents the English name of each number as text and randomly changes the starting index to give a diversity. Bimodal CelebA is a dataset of face images from the CelebA (Liu et al., 2015) with additional text describing the corresponding attribute labels. The PolyMNIST dataset consists of a set of five different MNIST images with different backgrounds and handwriting styles, each of which is considered a different modality. Different modalities of the same example have the same digits.
We confirm that the proposed model realizes the objective of Equation 5, i.e., it can properly infer representations and generate modalities from arbitrary subsets. To evaluate this quantitatively, we estimate the performance of the representation by training a linear classifier under inference from all modalities and predicting by it from the representation inferred from subsets. If this evaluation results in high performance on different subsets, it means that the inference from the subsets is successful. We also perform cross-generation from arbitrary modalities and evaluate the performance of the generated modalities by pre-trained classifiers from Sutter et al. (2020). 4 The interpretation of this evaluation is the same as in the case of representation.
For the network architectures, we follow Sutter et al. (2020), with 32-dimensional latent variables for MNIST-SVHN-Text and Bimodal CelebA and 256 for PolyMNIST. The distributions for images are set to Laplace, and Text is set to categorical. To align the scaling of the likelihoods for each modality, we divided each likelihood by the dimension of each modality and multiplied the largest dimension value by all likelihoods so that the coefficient of the likelihood term with the largest dimension was 1. We use Adam (Kingma & Ba, 2014) for optimization, with a learning rate of 0.001, and a mini-batch size of 256. We set β = 1, and train 50 epochs for MNIST-SVHN-Text and Bimodal CelebA and 300 epochs for PolyMNIST. All experimental results are the average of three runs. We implement them using PyTorch (Paszke et al., 2019) and Pixyz (Suzuki et al., 2021).

heading: 5.1 MNIST-SVHN-TEXT
text: Table 1 shows the results of the performance of representations in MNIST-SVHN-Text. MVAE has better performance for joint representations, but poorer performance for representations inferred from single modalities. MMVAE has better performance for inference from single modalities, but worse performance for joint inference. The performance of MoPoE compensates for the shortcomings of both methods. In contrast, the proposed method outperforms MoPoE in inference from
4https://github.com/thomassutter/MoPoE
most of the subsets, though it is lower than MMVAE for single modalities. This indicates that approximations that minimize the divergence from the surrogate posterior are more effective than computationally expensive approximations based on cross-generation. In addition, the performance of SMVGON is almost the same as that of SMVAE, which means that the surrogate posterior using VGON is sufficient as a joint inference model even without inference networks. See Appendix for the results of inference and generation when training with VGON alone. This result confirms that the performance of the surrogate posterior is sufficient for this dataset.
The results for generation in Table 2 are similar to those of the inference of the representation. It can be seen that the performance of the previous work on generating SVHN from subsets is greatly degraded. This is thought to be due to the difficulty in cross-generation to SVHN, which resulted in poor learning. On the other hand, the proposed method is able to generate from subsets with the same performance as from all modalities. This shows that the proposed method achieves our objective, i.e., to make inference from any modality as good as from all modalities.
Figure 1 shows the results of SVHN generation given different modalities for MoPoE-VAE, SMVAE, and SMVGON. We can see that MoPoE-VAE fails to generate SVHN from both MNIST and Text modalities. On the other hand, SMVAE can generate corresponding and diverse SVHN images from each modality. Sutter et al. (2020) attempted this generation by introducing modality-specific latent variables, but these results suggest that our proposed SMVAE can perform cross-generation of SVHN with only a single latent variable. In addition, it can be confirmed that SMVGON can generate images with similar performance, indicating that our objective can be achieved without inference networks.

heading: 5.2 BIMODAL CELEBA
text: Table 3 shows the results of representation and generation for Bimodal CelebA. On this dataset, SMVAE performs better than the other methods. We believe this is due to the challenging nature of this dataset. Although this dataset is bimodal, it has a larger diversity of images and text compared to MNIST-SVHN-Text. Therefore, it becomes difficult to learn reconstructions and cross-generations from a single modality, which makes previous methods that rely on this to learn inference insuffi-
cient. Our proposed method, on the other hand, learns the surrogate joint posterior by reconstructing all modalities, and approximates the inferences from arbitrary subsets to be close to it. Therefore, our method is less sensitive to the difficulty of cross-generation. On the other hand, SMVGON has not reached the same level of performance as SMVAE. This may be due to the difficulty of the multimodal dataset, which makes VGON’s inference insufficient. For the training results of VGON alone on this dataset, please refer to Appendix C. For qualitative generation results, see Appendix E.

heading: 5.3 POLYMNIST
text: Figure 2 shows the results of PolyMNIST. First, we can see that SMVAE outperforms the existing methods in both representation and generation. Also, the generation results show that SMVAE is able to generate corresponding numbers more clearly than MoPoE-VAE. This result confirms that SMVAE is a scalable model not only in terms of computational cost but also in terms of performance when the number of modalities increases. On the other hand, the performance of SMVGON is lower than that of other models in both generation and representation. This might be because, as in the case of bimodal CelebA, inference by VGON becomes difficult as the number of modalities increases.

heading: 5.4 CONCLUSION
text: In this study, we proposed SMVAE, which approximates inference from subsets by explicitly minimizing the divergence from the surrogate posterior distribution. The model is scalable with respect to the number of modalities and shows high performance in inference and generation, especially in the case where cross-generation is difficult. Furthermore, to reduce the parameters of the inference model, we introduced SMVGON using VGON, which can learn inference without inference networks. The remaining task of this study is to find a way to introduce a surrogate posterior distribution that will result in good inference with low memory and computational costs. VGON performed well on MNIST-SVHN-Text, but did not perform well on challenging problem settings such as Bimodal CelebA and PolyMNIST. In the future, we would like to address this issue and pursue effective methods for a wide range of multimodal datasets.

heading: 6 ETHICS STATEMENT
text: This work is a state-of-the-art method for multimodal VAEs, which will have an impact on the machine learning domain due to its scalability. Our method can cross-generate from one modality to another corresponding modality; therefore, we have to be aware that it can be used to generate non-existent facial images, text, spoken voices, etc. In addition, since the attributes in the CelebA dataset used for training in this study have biases (Prabhu et al., 2019), it is necessary to remove these biases when training the proposed method in practice.

heading: 7 REPRODUCIBILITY STATEMENT
text: All codes and datasets necessary to reproduce the experiments in this paper will be available after the paper review is concluded. The hyperparameters and other details in the training were written at the beginning of Section 5, and the architecture of the networks is the same as in Sutter et al. (2020).

heading: A COMPARISON BETWEEN SMVAE AND JMVAE WHEN THE NUMBER OF
text: MODALITIES IS 2
When the number of modalities is two, the objective function of JMVAE (Suzuki et al., 2016) is as follows:
L(Θ,Λ;X)− βDKL (qΛ(z|X)||qφ1 (z|x1))− βDKL (qΛ(z|X)||qφ2 (z|x2)) . (13)
Compared to Equation 11, the difference is that the direction of KL divergence is reversed and aggregation by PoE is not used. Table 4 shows the results of comparing SMVAE and JMVAE on the Bimodal CelebA dataset. The performance of the representations is almost the same, but the generation results are better for SMVAE. This may indicate that the KL divergence direction of SMVAE is better for cross-generation.

heading: B COMPARISON OF VAE AND VGON IN MNIST-SVHN-TEXT
text: Table 5 shows the inference and generation performance of VAE and VGON for multimodal inputs. It can be seen that VGON can generally learn inference models equivalent to VAE without inference networks, although its performance is slightly lower than that of VAE.

heading: C COMPARISON OF VAE AND VGON IN BIMODAL CELEBA
text: Table 6 shows the comparison between VAE and VGON in Bimodal CelebA. The results confirm that the performance of VGON is lower than that of VAE in all tasks.

heading: D EFFECTIVENESS OF END-TO-END LEARNING IN SMVAE
text: Figure 3 compares the results of end-to-end learning in SMVAE, i.e., optimizing Equation 11, and the results of two steps, i.e., first optimizing surrogate inference and then minimizing KL divergence with inference from arbitrary modalities. The dataset is PolyMNIST; in two stages, 200 epochs were trained in the first stage and 100 epochs in the second stage. The results show that the end-to-end method gives better results. This may be due to the fact that the two distributions are more easily approximated by training them simultaneously.

heading: E CONDITIONAL GENERATION ON BIMODAL CELEBA
text: Figure 4 shows the results of conditional generation of Bimodal CelebA given a text modality. We used SMVAE as the model.

heading: F RANDOM IMAGE GENERATION
text: Figure 5 shows the results of generating all modalities from random noise in the latent space for each dataset. We used SMVAE as the model. Each sample was generally generated correspondingly across modalities, but the text for Bimodal CelebA was not generated well.

Review summaries:
SMVAE aims to solve the challenge of combining multiple modalities into a coherent latent space using VAE framework. SMVAE is based on MoPoE-VAE model, the main architectural change is replacing the MoE rule with minimising the KL-divergence between the latent space distribution generated by the PoE combined experts and the joint encoder.    (The reviewer is sorry for the slight delay for submission as he is still on vacation and marked november third as deadline for submission. In the interest of time,the following is just a short summary of my notes)

This paper addresses the problem of formulating a well-justified ELBO for multi-modal VAEs.  The paper gives a thorough recounting of previous attempts at combining and training the per-mode posterior (approximations).  Previous workarounds involve, for instance, subsampling subsets.  This paper proposes approximating the joint posterior with a variational model and then adding the KLD to this variational posterior as an extra term on the ELBO, analogous with but an approximation to the KLD between the approximate and true posteriors that controls the gap in the ELBO.  The experiments report classification performance on MNIST + SVHN + Text and bi-modal CelebA.

In this paper, the authors propose a model for learning joint representations for multimodal data that allows inference when a few of the modalities are missing. Previous approaches addressed the missing modality problem by treating the inference network for each modality as an expert and combining the information across these experts. The proposed model claims to be useful in the following scenarios: 1) When cross-generation is difficult (and hence, MoPoE-VAE fails). 2) When the number of modalities is large.  The authors achieve this by learning a surrogate posterior conditioned on all the modalities while simultaneously optimizing the KL-Divergence between the surrogate posterior and a subset-conditioned posterior. The surrogate posterior is learnt by optimizing a variational lower bound to the log-likelihood objective. The authors evaluate the representations learnt by the model on secondary tasks and report improved performance. The generated modalities also perform well on secondary tasks. 

The authors propose a new approach to the class of scalable, multimodal VAEs. They propose to use a surrogate joint posterior which, using an additional sum of KL-divergence terms, helps keeping the posterior approximation of subsets of modalities similar to the joint posterior approximation. The proposed method is evaluated on two different datasets, MNIST-SVHN-Text and bimodal CelebA where the proposed method shows good results.