Scalable multimodal variational autoencoders with surrogate joint posterior
Authors: Masahiro Suzuki
Yutaka Matsuo
Venue: ICLR 2022 Submitted (2022)
Keywords: Deep generative models
multimodal learning

Abstract: To obtain a joint representation from multimodal data in variational autoencoders (VAEs), it is important to infer the representation from arbitrary subsets of modalities after learning.  A scalable way to achieve this is to aggregate the inferences of each modality as experts. A state-of-the-art approach to learning this aggregation of experts is to encourage all modalities to be reconstructed and cross-generated from arbitrary subsets. However, this learning may be insufficient if cross-generation is difficult. Furthermore, to evaluate its objective function, exponential generation paths concerning the number of modalities are required. To alleviate these problems, we propose to explicitly minimize the divergence between inferences from arbitrary subsets and the surrogate joint posterior that approximates the true joint posterior. We also proposed using a gradient origin network, a deep generative model that learns inferences without using an inference network, thereby reducing the need for additional parameters by introducing the surrogate posterior. We demonstrate that our method performs better than existing scalable multimodal VAEs in inference and generation.    


Under review as a conference paper at ICLR 2022
SCALABLE MULTIMODAL VARIATIONAL AUTOEN- CODERS WITH SURROGATE JOINT POSTERIOR
Anonymous authors Paper under double-blind review
ABSTRACT
To obtain joint representation from multimodal data in variational autoencoders (VAEs), is important to infer the representation from arbitrary subsets of modal- ities after learning. A scalable way to achieve this is to aggregate the inferences of each modality as experts. A state-of-the-an approach to learning this aggre- gation of experts is encourage all modalities to be reconstructed and cross- generated from arbitrary subsets. However, this learning may be insufficient if cross-generation is difficult. Furthermore, to evaluate objective function, expo- nential generation paths concerning the number of modalities are required. To al- leviate these problems, we propose to explicitly minimize the divergence between inferences from arbitrary subsets and the surrogate joint posterior that approxi- mates the true joint posterior. We also proposed using a gradient origin network, a deep generative model that learns inferences without using inference network, thereby reducing the need for additional parameters by introducing the surrogate posterior. We demonstrate that our method performs better than existing scalable multimodal VAFs in inference and generation.
INTRODUCTION
Our world characterized by many kinds of information. This multimodal information can be used to form common concepts. For instance, by recognizing information, such as the landscape of the beach and the salty scent of the ocean, we can form concept about the sea. In multi- modal learning (BaltruKaitis et al., (018)), it has been studied as the learning of joint representations from multimodal data, especially using variational autoencoders (VAEs) (Kingma Welling, 2013 Rezende al. 2014).
The goal of multimodal representation learning with VAEs (uzuki et al. 2016: Vedantam et al 2018: Wu & Goodman, 2018: (Shi et all 2019) is to learn all given multimodal data and to infer joint representation from arbitrary subsets of modalities. This representation is useful as an input to predict the corresponding label information as intermediate representation to generate other modalities, i.e., cross- geasssgenerations. However, this is difficult because a joint representation must be inferred when some of the modalities are missing test time. The most naive way deal with this difficulty is to have inference models for all possible combinations of modalities, which is exponentially necessary given the number of modalities.
A scalable way to address this is to consider the inferences for each modality as experts and combine them to approximate the inferences from arbitrary subset of modalities. There are two types of methods for aggregating inferences: products of experts (PoE) (Hinton, (202) and mixtures of experts (MoE): VAEs that use these approximate posteriors are proposed as MVAE Wu & Goodman, 2018) and MMVAE (Shi et all (219). respectively. Each method has dif- ferent advantages that exist in trade-off relationship. i.e., PoE increases the amount of mutual information with joint representation by increasing the number of modalities, while MoE has the effect of encouraging cross goos-genneration between modalities.
Sutter et al 1202 defined the objective of multimodal VAEs to be that the inference of all possi- ble subsets approximate the true joint posterior, and derived a generalized evidence lower bound (ELBO) as lower bound of that objective. This ELBO can be viewed using mixture of prod- uets of experts (MoPoE) as an aggregation method for inference, which is to generate all modalities from PoF-agggeeateed inference of arbitrary subsets. In other words, this model, called MoPoR-VAL

---

Under review as a conference paper at ICLR 2022
tries to achieve the above objective of multimodal VAEs by minimizing all reconstruction and cross- generation errors in all modality combinations. They showed that this is a generalization that has the advantages of both PoE and MoE. However, these properties of MoPoE-VAE imply that infer- ence may not be sufficiently learned when cross-generation is difficult. Furthermore, the number of paths for this reconstruction and cross-generation in MoPoE-VARE is required to be exponential with respect to the number of modalities. Therefore, we pursue a scalable method that realizes the above objective of multimodal VAES.
In this paper, we propose a different approach to achieve this objective. We first prepare a surrogate joint posterior, which approximates the true joint posterior given all modalities, and then minimize the divergence between the PoE-aggregated inferences and this surrogate posterior. The approx- imation of surrogate posterior is perform by the usual VAE optimization, and we simultaneously perform this optimization and the divergence minimization above.
The advantages of this new approach are twofold. First, if the surrogate model sufficiently close to the true posterior, then optimizing this objective function explicitly achieves the objective of multimodal VAEs defined MoPoE, i.e, approximating inferences from arbitrary subsets to the true posterior. Second, unlike MoPoE, this method is not exponentially more expensive. This is because this method is based on divergence minimization of all inference models, rather than reconstruction cross gessereneettinn of all modalities in combination. Although this method requires an additional surrogate model, the size is not exponential as in previous similar methods (Suzuki et all, 2016 Vedantam all 2018), but only doubles the number of parameters in encoders. Therefore, we call this method a scalable multimodal VAE (SMVAE).
Furthermore, to reduce the number of parameters in the surrogate posterior, we propose use variational gradient origin networks (VGONs) (Bond-Tavlor & Willcocks, 2020) for training the surrogate posterior. VGONs are deep generative models that can rapidly acquire amortized inference to representations without trainable network. We call our method that uses VGONs for surrogate posterior SMVGON.
Experiments show that the proposed method performs better than conventional methods in infer- ring joint representations and generating modalities on commonly used multimodal benchmarks. In particular, we show that the proposed method improves the performance in situations where the conventional methods have difficulty in cross-gent cross-generation. We also confirm that the performance of SMVGONs with VGONs is comparable to that of SM when the input modalities are relatively simple, without the need for additional inference networks for surrogate posteriors.
2 PRELIMINARY
2.1 MULTIMODAL VAES
Suppose that we are given an i..dd dataset (X'(0) 1 where each example is set of M modalities
( {xm (i) h m=1 and where each modality xm = (2( fimm.... Dmmm has its own dimension D,
and domain. We assume that these examples are derived from data distribution Pdtta (X) and that
each example x has corresponding common latent concept z [2( ie., a joint representation.
(6) E ER,
Given a training set {X our objective is to infer a joint representation from the subset of modalitic: X, C X and to generate another subset X C X via that representation. In other words, we aim to obtain an inference p(z[X) and a joint distribution p( p(X, z) -II.... e. (z)x(2))p(2) using a training set. In this study, we designate the generation when X, = X', as reconstruction and when X, n X 0 as cross-generation.
Rezende et al. 2014), which are deep gen- erative models that can learn inference in addition to generative models. Therefore, we can acquire representations of data by inference. The generative model (e (z) = I e Pe.,, (Xm (z), where θ {0m) m= is set of parameters, is parameterized by deep neural networks, and the prior of
To achieve this, we use VAES (Kingma & Welling, 2013:
We denote the multimodal set and any subset of it by X and Xs, respectively. We also omit the index of each example for brevity.
2

---

Under review as a conference paper at ICLR 2022
the latent variable is set as standard Gaussian p(z) IO. The objective function of VAEs maximize the marginal log-likelihood of the multimodal dataset: Σ Pe(;). However, since it is tractable to optimize this likelihood directly, we introduce an inference model qo (zlX) X) to approximate the posterior pe (z) , and instead optimize the following lower bound of the marginal log-likelihood given X, which is called the ELBO:
( Φ; X) Lg (x)x) [log pe [z)] - Dx (g)((2 (z) ppz)))
(1)
Here, the optimization of ELBO with respect to the parameters of the inference model corresponds minimizing the following Kullback-Leiber (KL) divergence:
DEL (4o (z) X (z)
In other words, maximizing the ELBO encourages bringing the true posterior closer to the inference model.
After optimizing Equation 1 aim to infer a joint representation z from arbitrary subset of modalities X., However, this is challenging because Equation only models joint inferences from all modalities. Therefore, for inferences from subset of modalities X,, other modality inputs missing, which results in the inability to infer the representation properly (Suzuki et all 2016).
A scalable way to alleviate this difficulty is to consider the inference model for each modality an expert and approximate the inference model for arbitrary modalities by combining them. There are two ways combine experts: product and mixture.
2.1.1 AGGREGATED INFERENCE WITH POE AND MoE
A joint inference given all modalities X is expressed the following using PoE (Hinton. 2002):
(roe(z|X) x p(z) II Xm EX
4bm. (z(xm),
(3)
In general, PoE cannot be calculated as closed-form, but if we restrict each expert to Gaus- is calculable as (zoE(z|X)) =
sian 4om (z|xm) = A' (Hm, diag(o? ?))), then Equation 3
NH, diag())))) where H4 En Hymo jm and of Σn 7mm enables us to infer the representation from subset X,.
Therefore, the Gaussian PoE
Σm 7.m.
However, if we optimize this inference model with Equation 1 then the inference of arbitrary modalities is not optimized explicitly. Wu & Goodman (2018) proposed to train ELBOs with PoÉ- aggregated inferences of all modality combinations Σx Ex.x L(O,, t.; X²); however, since the number of ELBOs increases exponentially, they used the subsampled version. The model that learns PoE inference by this objective function is called MVAE (Wu & Goodman. 2018).
Another method to combine experts to use MoE:
aMor (Mox(
Σ Wmlemm (z Xm),
Xm
where Wm is mixing coefficient of each expert and where Σ Ex Wm = 1. Shi et al (2019) proposed VAE with MoE-aggregated inference models called MMVAE.
There is trade-off between the approximate posterior with PoE and MoE. PoE can perform infer- ence given any subset and MVAE can learn inference explicitly during training. PoE also has the property that increasing the size of the subset, i.e., the number of input modalities, leads to an in- crease the information content of the inferred latent variables. However, the objective function of MVAE does not guarantee that the inferences from subset are the same, since it only Iearns construction and not cross-generaiie cooss-enneration Sutter et al. 2021) also pointed out that its objective function by subsampling is not valid lower bound.
MMVAE based on MoE, on the other hand, has a valid ELBO and learns to explicitly encourage cross gooss-ennraatin. which ensures that the inferences from each modality remain consistent. However, this method only learns inferences from a single modality, and increasing the number of modalities does not necessarily increase the information content of the representation.
3

---

Under review as a conference paper at ICLR 2022
2.1.2
MoPoE- VAE
Sutter et (2021) considered optimization on all possible subsets, i.e., the powerset P(X), and reestablished the minimization of the following KL divergence as the objective of multimodal VAEs:
arg min
Σ X.EP(X)
DKL (go. (a|X)) (z)
(5)
They then proposed a generalized ELBO that is a valid lower bound on the log-likelihood pe (X) and whose maximization corresponds to the minimization of Equation
(e)))) [log pe (X [z)] (xs.
(6)
where the form of the posterior of @MoPoE is called MoPoE and is defined follows:
[MoPoE(Z|X)
1
Σ
2M
X2EP(X)
A feature of the model that maximizes Equation[ called MoPoE- VAE, is that it learns to generate all modalities from all possible subsets, including reconstruction and cross gooss-ennrattin, which improves the performance of inference and generation from all subsets. In addition, they showed that this MoPoE-VAE a generalization of MVAE and MMVAE, specifically MVAE when the summation in Equation is taken only for a of all modalities X and MMVAE when it is taken for the single modalities {xm LM m= Therefore, MoPoE-Van VAE can be regarded as having the advantages of both MVAE and MMVAE.
2.2 VARIATIONAL. GRADIENT ORIGIN NETWORKS
VGONs BBono-Taylr & Willcocks. 2020) deep generative models that can perform amortized inference equivalent to VAEs without an inference network
Given an observed variable x and latent variable z, let the generative model be pa (x(z), and the inference model be 40.2, (z) = N (zo)) (z(2)) with the mean and variance mapped by trainable linear transformations (pi and oz) from parameter Zg. This objective function, ELBO, given X is defined as follows:
Lv o, zo; x0;x) =E (o - D (go.na (z)lp(()))
In VGONs, the estimator for zo calculated by a single update based on the gradient Equation
20 2 2o LVGON (0, Φ, 20; 20;x).
where 2o 0, i.e, the initial value of Zo is set to the origin. Using this estimator 2n. we can obtain inference as ~ 20.20 (z) and can evaluate the objective function as LvGoN (0, o, 2o; x). In learning, we also backpropagate the gradient of the objective function through Equation D which allowed us to learn both inference and generation simultancously as in VAEs.
METHODS
3.1 ISSUES OF MOPOE- VAE
MoPoE-VAL aims the objective of Equation b maximizing the ELBO of Equation 6 The two are equivalent in variational inference, but have different implications for learning including genera- tive models, i.e., whether the Equation[]js achieved depends the performance of all modalities of reconstruction and cross- cross-generation. Therefore, in situations where cross-generation is difficult, this objective might be fully achieved, specifically, the inference and generation of some modalities might be significantly worse than others.
In the orig original paper (Bond-Taylos & Willeocks 2020), gradient origin networks were introduced first, and then variational GONs and implicit GONs introduced an extension of them.
4

---

Under review as a conference paper at ICLR 2022
Another issue is that the computational cost of evaluating Equation 6 increases exponentially with the number of modalities. Specifically, we consider a computational path that generates modalities from inferred representations. In this case, since there are 2M representations to be inferred by MoPoF, the generation path required to evaluate Equation + is x 2M; therefore, as the number of modalities increases, the evaluation of the ELBO becomes more intractable and might also lead to insufficient learning of cross- tross-generation.
3.2 A SURROGATE, JOINT POSTERIOR AND THE REFORMULATION OF THE OBJECTIVE
To address the above issues, we aim to explicitly realize Equation 5 However, since a true posterior cannot be obtained, we prepare surrogate joint posterior 9A (x),,), which is Gaussian distri- bution parameterized by neural network and should approximate the true posterior pe (z) X). We then reformulate our objective minimizing the KL divergence between subset inferences and this surrogate posterior:
arg min
Σ X.EP(X)
DKL. (as. (x)Xx) || (z)
(10)
Here, we consider each inference from a subset qa. (x) to be expressed in PoE form. Then, since the surrogate posterior is defined as Gaussian, this KL divergence can be calculated analytically.
X)) Ra is satisfied, then this optimization problem the same as Equa- tion 5 As explained in Section 2.1 minimizing the KI. divergence between the approximate pos- terior (in this case, the surrogate posterior) and the true posterior is equivalent to maximizing the ELBO. Therefore, we can first maximize the ELBO L(O, A; X') sufficiently, and then fix the sur- rogate posterior and perform the optimization of Equation IO However, this requires a two-step learning process, which can be time consuming. Therefore, we propose to optimize these two steps simultaneously, referring to existing multimodal VAES with additional inference models such as JMVAE (Szuuii al. 2016) (see Section 4) i.e., maximize the following objective function:
If D (qa (z) X)
((),A;
B 2M
DRL (gor (z)Xx) || (x)x)), X.EP(X)
Σ
(11)
where 9 is hyperparamettrr that adjusts for the effect of KL divergence in the objective function. We train this objective function in an end-to-end fashion. Empirically, we have found that this end-to- end optimization results in higher performance than the two-stage optimization (see Appendix D) At test time, we infer representations from subsets using PoE-aggregated inference model qs. Xx).
The advantages of this objective function are as follows. First, inference models from arbitrary subsets are learned not by cross-generation, but by explicitly minimizing the divergence between them and the surrogate posterior. Moreover, the surrogate posterior is learned based on the maxi- mization of ELBO Z(O, A; X), i.e., based on the reconstruction in all modalities, rather than cross- generation. Therefore. this objective can mitigate the impact of the difficulty of cross-generation on the performance of inference and generation.
Secondly, the computational cost can be significantly reduced compared to MoPoE- MAE. There is only one pass for generation in evaluating this objective function for each modality in Z(O, A; X). In addition, the second term of [qqutiio[[[]]omppl obtained by aggregating the inferences for each modality; therefore, the inference pass is also only once for each modality.
the inference for all possible subsets, which is
We call the proposed model a scalable multimodal VAE (SMVAE) because is scalable to the number of modalities while explicitly optimizing toward the objective of multimodal VAEs.
3.3 REDUCING THE PARAMETERS IN SURROGATE INFERENCE MODELS
Although the proposed method alleviate the issues of previous multimodal VAEs, it also has the disadvantage of requiring more parameters for the inference model. The added surrogate posterior requires as many parameters all of the unimodal inference models, i.e., the parameter size of the inference model is twice as large as that of previous models such as MoPoE- VAE. Note that in ex- isting studies of multimodal VAEs with additional inference models, such as JMVAF, the parameter
5

---

Under review as a conference paper at ICLR 2022
size increases exponentially because inference models required for each subset. In comparison, SMVAE keeps the number of required parameters linear with respect to the number of modalities.
To reduce the number of required parameters, we propose to VGONs to train the surrogate posterior and generative models for all modalities. In the VGON framework, the surrogate posterior given multimodal inputs can be defined as ax.zo (z) (zx( (z,) where the estimator of 2o is calculated as 20 = Z Lvco (O, X, Zo: X). Therefore, the objective function of the proposed method using VGONs as follows:
B 2
LYGON LvGoN(O,A, Zo:
Σ XxEP(X)
DKL (gs. (z|Xe) 11 (z)).
(12)
We refer to this model as SMVGON to distinguish it from SMVAE
Note that the surrogate posterior is used to approximate the aggregated inference model during train- ing and is not used during testing; therefore, is possible to use other iterative inference algorithms. In this study, however, we used VGONs, which perform inference in single iteration, to avoid slow training.
RELATED WORKS
For learning unsupervised joint representations from multimodal data, autoencoder- methods have been used (Neiam et al., 2011 Silberer & Lapata. 2014); however, they have difficulties in the challenge of missing modalities. Joint representation learning using deep belief networks (Hinton 2009) or deep Boltzmann machines (Salakhutdinov & Hinton, 2009) can adequately handle missing modalities (Srivastava & Salakhutdinov, 2012: Srivastava et all 2012 Sohn et all (2014), but has difficulty scaling to large dataset.
Early multimodal learning using VAES dealt with cross- cross-generation of two modalities, such as conditional VAES (Sohn et all 12015: Kingma al 2014) and conditional multi-modal autoen-
coders (Pandey & Dukkipati (017)) However, because these methods directly learn conditional generative models, they cannot perform bidirectional generation between modalities, nor can they obtain joint latent representations of different modalities.
Some studies of multimodal VAES require additional inference models (Suzuki al 2016 Vedantam et all (2018: Korthals et all 2019: Wu & Goodman 2019). Suzuki et al. (2016) pointed out that VAEs trained on all modalities have the problem of missing modalities
inference, and propose to approximate the unimodal inference models to explicitly approach the joint inference model of VAES. The objective function of JMVAE the ELBO of VAE given multimodal inputs minus the KI. divergence corresponding to the above approximation! Vedantam et al. (2018) g) intro- duced TELBO, where the objective function is composed of the sum of ELBOs that take each set of modalities imput, using the same models as JMVAE. Korthals et all (019)) introduced VAE that can be regarded as a combination of JMVAE and TELBO, and Wu Goodman (2019) pro- posed VAEVAE that excludes KL divergence term between joint inference and prior from it. They are learned end-to-end with single objective function, except for TELBO.
The challenge with these models is that they require inference models for all modality combina- tions, i.e., the number of inference models grows exponentially with the number of modalities. Our proposed method is similar these methods; however, unlike these methods, required memory cost is linear in the number of modalities. Kutuzova et al. (2021) proposed to introduce additional uni- modal inference models instead of joint inference model, and to reduce the number of required inference models to the square of the number of modalities. However, this also does not scale with the number of modalities compared to our linear cost model.
MMJSD proposed by Sutter et al. (2020) is closer to our work, i.c., it encourages unimodal infer- ence to be closer to joint inference from modalities. The differences between our work and
When the number of modalities is two, our proposed method is similar to JMVAE, except that the direction of KL divergence is reversed and PoE-aggregation is used for inference. Please refer to Appendix for the similarity and performance difference hetween JMVAR and SMVAF. When the number of modalities is two, the performance of the representations is comparable, but SMVAF outperforms in the cross-gencration.

---

Under review as a conference paper at ICLR 2022
MMJSD are twofold. First, while MMJSD only considers unimodal inference, work approx- imates joint inference with any combination of unimodals, which allows to learn inferences from arbitrary modalities. The second is the design of joint inference. MMJSD considers dynamic prior that combines all unimodal inferences and considers it a joint inference. However, there is guarantee that this dynamic prior approximates the true posterior in the surrogate posterior of SMVAE. In addition, since MMJSD uses the same networks for a joint inference and unimodal inferences, might be difficult for this joint inference to achieve all of the approximations to the true posterior and to inferences from arbitrary modalities.
The other topic in multimodal VAEs is the acquisition of -ppecif latent variables. apart from the global latent variables that embed all modalities (Tsai et al., 2018; [ssu & Glass, [2018
Sutter et al 2020). Our study is in different direction from these, since we are addressing the possibility of inferring a single joint representation from arbitrary subset of modalities. In the exper- iments, we confirm that SMVAE achieves high performance with single latent variable task where the conventional method achieves by adding modality-speciffe latent variables.
Recently, contrastive-based multimodal (multi-view) learning TTan et al 2019 Alavrac et ald 2020; [Taa et alL (202)) has been gaining attention instead of autoencoder-based but these aim to acquire coordinated representations (Baltrušaitis et al. 2018). i.c., they do not assume inference from subset of arbitrary modalities.
EXPERIMENTS
We use MNIST-SVHN-Tex Sutter et al. 2020), Bimodal CelebA (Sutter et al. 2020), and PolyMNIST (Sutter et al. 102 datasets for evaluation. MNIST-SVVNN-TX is the MNIST- SVHN (Shi et al., 2019) with the addition of the text modality, which represents the English name of each number text and randomly changes the starting index to give diversity. Bimodal CelebA is dataset of face images from the CelebA (Liu et all (2015) with additional text describing the corresponding attribute labels. The PolyMNIST dataset consists of set of five different MNIST images with different backgrounds and handwriting styles, each of which is considered a different modality. Different modalities of the same example have the same digits.
We confirm that the proposed model realizes the objective of Equation 5 i.e., it can properly infer representations and generate modalities from arbitrary subsets. To evaluate this quantitatively, we estimate the performance of the representation by training linear classifier under inference from all modalities and predicting by it from the representation inferred from subsets. If this evaluation results in high performance on different subsets, it means that the inference from the subsets is suc- cessful. We also perform cross- sessseenerttion from arbitrary modalities and evaluate the performance of the generated modalities by pre-trained classifiers from Sutter et al. (020). The interpretation of this evaluation is the same as in the case of representation.
For the network architectures, we follow Sutter all 2020). with 32-dimensional latent variables for MNIST-SVHN-TE and Bimodal CelebA and 256 for PolyMNIST. The distributions for images set to Laplace, and Text is set to categorical. To align the scaling of the likelihoods for each modality, we divided each likelihood by the dimension of each modality and multiplied the largest dimension value by all likelihoods so that the coefficient of the likelihood term with the largest dimension was 1. We use Adam (Kingma & Ba, (0044) for optimization, with a learning rate of and
0.001, and mini-batch size of 256. We set 3 = 1, and train 50 epochs for MNIST-SVHN- Bimodal CelebA and 300 epochs for PolyMNIST. All experimental results are the average of three runs. We implement them using PyTorch (Paszke et all, 2019) and Pixyz (Suzuki al., 2021).
5.1 MNIST-SVHN-TE
Table shows the results of the performance of representations in MNIST-SVHN MVAE has better performance for joint representations, but poorer performance for representations inferred from single modalities. MMVAE has better performance for inference from single modalities, but worse performance for joint inference. The performance of MoPoE compensates for the shortcom- ings of both methods. In contrast, the proposed method outperforms MoPoE in inference from

---

Under review as a conference paper at ICLR 2022
Table I: Classification accuracy the latent representation in MNIST-SVHN--TeN Bold represents
highest accuracy each modality.
MODEL
S
T
M+S
M+T
S+T
M4S+T
MVAE
0.82
0,43
0.85
0,86
0.98
0.82
0.98
MMVAE
0.97
0.83
0.99
0,90
0.99
0.91
0.93
MoPOE-VAE
0.88
0.75
0.99
0.91
0,98
0.93
0.98
SMVAE
0.95
0.78
0.99
0.96
0.99
SMYGON
0.96
0.80
0.99
0.98
0.99
0.99
0.99
Table 2: Classification accuracy of image generations MNIST-SVHN- Text. The top line shows modalities generated, and the second line shows modalities conditioned when generating it.
M
MODEL
S+1
M+S+T
M
M-1
M-S+1
M+S
M+S+1
MVAE
0.23
0.17
0.27
0.95
0.44
0.37
0.65
0.78
0.97
0.19
0.25
0.99
MMVAB
0.99
0.91
0.92
0.36
0.38
0.37
0.45
0.97
0.82
0.89
0.93
MOPOE-VAE
0.79
0.99
0.94
0.96
0.34
0.35
0.35
0.85
0.97
0.81
0.98
0.99
SMVAE
0.72
0.99
0.97
0.97
0.74
0.82
0.77
0.90
0.94
0.74
0.95
0.99
SMYGON
0.98
0.99
most of the subsets, though it is lower than MMVAE for single modalities. This indicates that approximations that minimize the divergence from the surrogate posterior are more effective than computationally expensive approximations based on cross-generation. In addition, the performance of SMVGON almost the same as that of SMVAE, which means that the surrogate posterior using VGON is sufficient joint inference model even without inference networks. See Appendix for the results of inference and generation when training with VGON alone. This result confirms that the performance of the surrogate posterior is sufficient for this dataset.
The results for generation Table are similar to those of the inference of the representation. be seen that the performance of the previous work generating SVHN from subsets is greatly degraded. This is thought to be due to the difficulty in cross- goosssegneration to SVHN, which resulted in poor learning. On the other hand, the proposed method able generate from subsets with the same performance as from all modalities. This shows that the proposed method achieves our objective, i.c., make inference from any modality as good as from all modalities.
Figure I shows the results of SVHN generation given different modalities for MoPoE-VAE, SM- VAE, and SMVGON. We can see that MoPoE-VAD fails to generate SVHN from both MNIST and Text modalities. On the other hand, SMVAE can generate corresponding and diverse SVHN images from each modality. [Sttter al. (200)) attempted this generation by introducing modality-speciff latent variables, but these results suggest that our proposed SMVAE can perform cross-generation of SVHN with only a single latent variable. addition, can be confirmed that SMVGON can generate images with similar performance, indicating that our objective can be achieved without inference networks.
5.2 BIMODAL CELEBA
Table shows the results representation and generation for Bimodal CelebA. On this dataset, SMVAE performs better than the other methods. We believe this is due to the challenging nature of this dataset. Although this dataset is bimodal, it has a larger diversity of images and text compared to MNIST-SVHN--E Therefore, it becomes difficult learn reconstructions and cross-gencd coss-generations from a single modality, which makes previous methods that rely on this to learn inference insuffi-
MoPoE-VAE
SMIAAE
SMYGON
MNIST
0 3 / 1 S 9
4
0
3
/
5
9
4
0
3
/
1
5 9
4
SVHN
Figure 1: Cross generatinnn of SVHN on MoPoE, SMVAE, and SMVGON. Each row of SVHN in each cross-generation corresponds to different sample from the inference model.
8

---

Under review as a conference paper at ICLR 2022
Table 3: Performance the latent representation and modality generation in Bimodal CelebA, and respectively stand for Image and Text. We report the mean average precision across all modalities.
LATENT REPRESENTATION
GENERATION
T
MODEL
T
I+T
T
[+
[+1
MVAE
0.48
0.39
0.54
0.30
0.51
0.25
0.50
MMVAE
0.51
0.48
0.50
0.45
0.45
0.34
0.34
MoPOE-VAL VAE
0.51
0.43
0.53
0.42
0.50
0.32
0.40
SMVAE
0,56
0.56
0.59
0,46
0.53
0.36
0.50
SMVGON
0.50
0.56
0.52
0.47
0.45
0.32
0.45
Accuracy of latent represenationn
1.00
M0
f
B24
H1
0.95
0.90
0.8
Molol-VRR
0.85
06
Accur of
MI, M2 M4
MACE
MI ML. M2,
0.75
SMIVE
N0. ML 501
070
Number input modalities
Number of Input modalities
MCC, M4
Figure 2: Performance the PolyMNIST dataset. Left: Performance of the inferred latent repre- sentation given a set of modalities for each number. The markers are the means of the three runs, and the error bands their standard deviations. Center: Performance of generating modalities given set of modalities corresponding to each number. Right: Conditional generation of PolyMNIST.The top panel shows examples of modalities MO to M3 (we refer the five different modalities MO to M4), and the middle and bottom panels show the generation of M4 conditioned on the respective modality sets by MoPoE-VAE and SMVAE. Images each column belong the same example.
cient. Our proposed method, the other hand, learns the surrogate joint posterior by reconstructing all modalities, and approximates the inferences from arbitrary subsets to be close to it. Therefore, method is less sensitive to the difficulty of cross poss-generation. On the other hand, SMVGON has not reached the same level of performance as SMVAE. This may be due to the difficulty of the multimodal dataset, which makes VGON's inference insufficient. For the training results of VGON alone on this dataset, please refer to Appendix C For qualitative generation results, see Appendix
5.3 POLYMNIST
Figure 2 shows the results of PolyMNIST. First, we can see that SMVAE outperforms the existing methods both representation and generation. Also, the generation results show that SMVAE able to generate corresponding numbers more clearly than MoPoE-VAE. This result confirms that SMVAE is scalable model not only in terms of computational cost but also in terms of performance when the number of modalities increases. On the other hand, the performance of SMVGON is lower than that other models in both generation and representation. This might be because, as in the case of bimodal CelebA, inference by VGON becomes difficult as the number of modalities increases.
5.4 CONCLUSION
In this study, we proposed SMVAE, which approximates inference from subsets by explicitly mini- mizing the divergence from the surrogate posterior distribution. The model is scalable with respect to the number of modalities and shows high performance in inference and generation, especially in the case where cross cross-generation is difficult. Furthermore, to reduce the parameters of the infer- ence model, we introduced SMYGON using VGON, which can learn inference without inference networks. The remaining task of this study is to find way to introduce surrogate posterior dis- tribution that will result in good inference with low memory and computational costs. VGON per- formed well KNIST-SVHN-TeH as Bimodal CelebA and PNIST. In the future, we would like to address this issue and pursue effective methods for wide range of multimodal datasets.
but did not perform well on challenging problem settings such

---

Under review as a conference paper at ICLR 2022
6 ETHICS STATEMENT
This work is a state-of-the-art method for multimodal VAES, which will have an impact on the machine learning domain due to its scalability. Our method cross-generafs from modality to another corresponding modality; therefore, we have to be aware that it can be used to generate non-existent facial images, text, spoken voices, etc. In addition, since the attributes in the CelebA dataset used for training in this study have biases (Prabhu et al., 2019), it is necessary to remove these biases when training the proposed method in practice.
REPRODUCIBILITY STATEMENT
All codes and datasets necessary reproduce the experiments in this paper will be available after the paper review is concluded. The hyperparameters and other details in the training were written at the beginning of Section and the architecture of the networks is the same as in Sutter al. (2020).
REFERENCES
Jean-Baptiste Alayrac, Adrià Recasens, Rosalia Schneider, Relja Arandjelović. Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-supervised mul- timodal versatile networks. arXiv preprins arXiv:2006. 16228, 2020.
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philipee Morency. Multimodal machine learning: A survey and taxonomy. IEEE transactions pattern analysis and machine intelligence, (2): 423-443, 2018
Sam Bond- Taylor and Chris G Willcocks. arXiv:2007.02798, 2020.
Gradient origin networks.
arXiv preprint
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 1778)1777- 2002.
Geoffrey E Hinton. Deep belief networks. Scholarpedia, 4(5):5947, 2009.
Wei-Ning Hsu and James Glass. Disentangling by partitioning: A representation learning framework for multimodal sensory data. arXiv preprint arXiv:1805.1 2018
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Diederík Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Diederik Kingma, Danilo J Rezende, Shakir Mohamed, and Max Welling, Semi-supervised learn- ing with deep generative models. arXiv preprint arXiv:1406.5298 2014.
Timo Korthals, Daniel Rudolph, Jürgen Leitner, Mare Hesse, and Uirich Rückert. Multi-modal generative models for learning epistemic active sensing. In 2019 International Conference on Robotics and Automation (ICRA), pp. 3319-3325. IEEE, 2019.
Svetlana Kutuzova, Oswin Krause, Douglas McCloskey, Mads Nielsen, and Christian Igel. Mul- timodal variational autoencoders for semi-supervised learning: In defense product-of-experts. arXiv preprint arXiv:2101.07244. 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang. and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lec, and Andrew Y Ng. Multi- modal deep learning. In ICML, 2011.
Gaurav Pandey and Ambedkar Dukkipati. Variational methods for conditional multimodal deep learning. 2017 International Joint Conference on Neural Networks (IJCNN), pp. 308-315. IEFE, 2017.
10

---

Under review as a conference paper at ICLR 2022
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Vinay Uday Prabhu, Dian Ang Yap, Alexander Wang, and John Whaley, Covering up bias in celeba- like datasets with markov blankets: A post-hoc cure for attribute prior avoidance. arXiv preprint arXiv: /907. 12917, 2019.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International conference machine learning, pp. 1278-1286. PMLR, 2014.
Ruslan Salakhutdinov and Geoffrey Hinton. Deep boltzmann machines. In Artificial intelligence and statistics, pp. 448 448.455. PMLR, 2009.
Yuge Shi, Narayanaswamy Siddharth, Brooks Paige, and Philip Torr. Variational mixture-of-exxeeee autoencoders for multi-modal deep generative models. In Advances in Neural Information Pro- cessing Systems, pp. 15718-15729, 2019.
Carina Silberer and Mirella Lapata. Learning grounded meaning representations with autoencoders. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 721-732, 2014.
Kihyuk Sohn, Wenling Shang, and Honglak Lee. Improved multimodal deep learning with variation of information. Advances in neural information processing systems, 27:2141-2149, 2014.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28: 3483-3491, 2015.
Nitish Srivastava and Ruslan Salakhutdinov. Learning representations for multimodal data with deep belief nets. In International conference machine learning workshop, volume 79, pp. 3, 2012.
Nitish Srivastava, Ruslan Salakhutdinov, al. Multimodal learning with deep boltzmann machines. In NIPS, volume 1, pp. 2. Citeseer, 2012.
Thomas M Sutter, Imant Daunhawer, and Julia E Vogt. Multimodal generative learning utilizing jensen-shamooon-iveeggeeee arXiv preprint 2XX:22006.08242, 2020.
Thomas M Sutter, Imant Daunhawer, and Julia Vogt. Generalized multimodal elbo. arXiv preprint arXiv:2/05.02470, 2021.
Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Joint multimodal learning with deep generative models. arXiv preprint arXiv:1611.01891, 2016.
Masahiro Suzuki, Takaaki Kaneko, and Yutaka Matsuo. Pixyz: a library for developing deep gener- ative models. arXiv preprint 1011 2021.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint 1rXiv:1966.05849, 2019.
Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhut- dinov. Learning factorized multimodal representations. arXiv preprint arXiv:/806.06. 066006666666666 2018.
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Self- supervised learning from multi-view perspective. In International Conference Learning Representations, 2021.
Ramakrishna Vedantam, Ian Fischer, Jonathan Huang, and Kevin Murphy. Generative models of visually grounded imagination. In International Conference Learning Representations, 2018 8.
11

---

Under review as a conference paper at ICLR 2022
Mike Wu and Noah Goodman. Multimodal generative models for scalable weakly-supervinn learn- ing. In Advances in Neural Information Processing Systems, pp. 3575-55855 2018.
Mike Wu and Noah Goodman. Multimodal generative models for compositional representation learning. arXiv preprint arXiv:1912.05077. 2019.
A COMPARISON BETWEEN SMVAE AND JMVAE WHEN THE NUMBER OF MODALITIES IS 2
When the number of modalities two, the objective function of JMVAE (Szuui al. 2016) is as follows:
(().A:X) - -DDL (ga(z)X))lg,, (z|x1)) - 3DKL (ga(z|X))lao, (z(x²)).
(13)
Compared to Equation I the difference that the direction of KL divergence is reversed and aggregation by PoE is used. Table shows the results of comparing SMVAE and JMVAE on the Bimodal CelebA dataset. The performance of the representations is almost the same, but the generation results are better for SMVAE. This may indicate that the KI. divergence direction of SMVAE is better for cross- cross-generation.
B COMPARISON OF VAE AND VGON MNIST-SVHN-TEX
Table Sishows the inference and generation performance of VAE and VGON for multimodal inputs. can be seen that VGON can generally learn inference models equivalent to VAE without inference networks, although its performance is slightly lower than that of VAE.
C COMPARISON OF VAE AND VGON IN BIMODAL CELEBA
Table O shows the comparison between VAE and VGON in Bimodal CelebA. The results confirm that the performance of VGON lower than that of VAE in all tasks.
D EFFECTIVENESS OF END-TO-END LEARNING IN SMVAE
Figure compares results of end-to-end learning in SMVAE, i.e., optimizing Equation and the results of two steps, i.e., first optimizing surrogate inference and then minimizing KL divergence with inference from arbitrary modalities. The dataset is PolyMNIST; in two stages, 200 epochs were trained in the first stage and 100 epochs in the second stage. The results show that the end-to-end method gives better results. This may be due to the fact that the two distributions are more easily approximated training them simultaneously.
E CONDITIONAL GENERATION ON BIMODAL CELEBA
Figure 4 shows the results of conditional generation of Bimodal CelebA given modality. We used SMVAE as the model.
Table 4: Performance on the latent representation and modality generation in Bimodal CelebA.
Latent representation
Generation
I
T
MODEL
I
T
1+T
1+1
I+T
JMVAE
0.56
0.56
0.59
0.54
0.30
0.49
SMVAE
0.58
0.56
0.59
0.46
0.53
0.36
0.50
12

---

Under review as a conference paper at ICLR 2022
Table 5: Classification accuracy on the latent representation and generated modalities in MNIST- SVHN-Text. M, S and respectively stand MNIST, SVHN, and Text, indicating that the rep- resentations inferred from these modalities. M+S means both M and S. A-> B means that the corresponding column is the performance of generating B conditioned on A.
LATENT REPRESENNATION
GENERATION
MODEL VAE
M4S+T
M+S+T + M
M4S+T-+S
M+S+T-T
0.99
0.96
0.99 0.87
VGON
0.99
0.91
0.85
0.99
Table 6: Classification accuracy on the latent representation and generated modalities in Bimodal CelebA.
LATENT REPRESENIATION
GENERATION
MODEL VAE
I+T
1+T +1
I+T-T
0.59
0.97
0.50
VGON
0.57
0.92
0.45
F RANDOM IMAGE GENERATION
Figure shows the results generating all modalities from random noise in the latent space for each dataset. We used SMVAE the model. Each sample generally generated correspondingly across modalities, but the text for Bimodal CelebA was not generated well.
Accuracy of latent representation
1.00
0.90
SMVAE (end-to-end)
Accuracy image generation
SMVAE (two stages)
0.98
0.96
0.94
0.92
0.75
0.90
0.70
0.88
SMVAE (end-to-end)
0.65
0.86
SMVAE stages)
Number of Input modalities
Number of input modalities
Figure 3: Performance on the PolyMNIST dataset. Left: Performance of the inferred latent rep- resentation given set of modalities for each number as input. Right: Performance of generating modalities given a set modalities corresponding to each number.
13

---

Under review as a conference paper at ICLR 2022
Weed tar, fith bany caling Bukk chulhhoe pon hony thikhoos on boaks
Sia cycreea rald
noutiilgee pertrooss Ladiigg 190
collenaaoo herad polarian
preae nol (rO WEudR
baps snder cy hard
Text
peary anliig your haaine
Krvloves Snoutin
-Image
Figure Conditional generation Bimodal CelebA on SMVAE.
7
7
5 3 4
6
S
6
9
4
#
5
8
Fige
zesr
eight
eight
1
2
Q
6
2
7
?
7
3
3
7
5
/
f
!
2
f
J
7
5
7
5
3
2
I
9
J
/
7
S
g
3
7
7
2
2
/
4
/
?
/
2
7
5
7
5
2
/
2 7
Figure 5: The results of generating all modalities from random noise in the latent space on each dataset. Top: MNIST-SVHN-Tex (from top bottom: MNIST, SVHN, Text). Center: Bimodal CelebA (Image Text). Bottom: PolyMNIST (MO, MI, M2, M3, M4). Each column of each dataset corresponds to the same random noise.
14


Reviews:
[{"review_id": "v9jYVvRTlz7", "reviewer": "Reviewer_yNag", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Perhaps it is because I am not an expert in multi-modal VAEs, but I found this paper quite hard to read.  It presents to the reader a series of models, variational formulations, and ELBOs, and I had a hard time tracking the relationships between each.  My primary critique then is on the paper's presentation of its innovation, which takes form in Equations 10 and 11.  I really wanted to see a derivation that connects equations 10 and 11.  Since there are variational models on both sides of Equation 10, it's not clear to me how the (generative) model joint $P(X, Z)$ arises in the ELBO presented in Equation 11.  Relatedly, I found the paper's claims about the proposed framework, such as \"it is unlikely that the performance of inference and generation will be degraded by the difficulty of cross-generation\" (p 6) to be hard to agree with or verify as these concepts are not defined with any rigor.", "summary_of_the_paper": "This paper addresses the problem of formulating a well-justified ELBO for multi-modal VAEs.  The paper gives a thorough recounting of previous attempts at combining and training the per-mode posterior (approximations).  Previous workarounds involve, for instance, subsampling subsets.  This paper proposes approximating the joint posterior with a variational model and then adding the KLD to this variational posterior as an extra term on the ELBO, analogous with but an approximation to the KLD between the approximate and true posteriors that controls the gap in the ELBO.  The experiments report classification performance on MNIST + SVHN + Text and bi-modal CelebA.", "summary_of_the_review": "The paper's presentation---specifically, the lack of a derivation that connects Eqs 10 and 11---makes it very hard for me to assess the paper's contributions.  Thus, I think the paper needs a major revision to improve clarity.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lyp2OizwpSN", "reviewer": "Reviewer_FDzr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- Interesting approach to scalable, multimodal VAEs. It seems to be a reasonable trade-off between computational complexity and memory-constraints.\n- Disregarding some typos and missing information (see minor comments), clear description of the proposed method\n\n### Weaknesses\n- For empirical evaluation, quantitative results on the quality of generated samples are missing as well as classification results of randomly generated samples (see Shi et al, 2019). Also, there are no results reported with respect to test set log-likelihoods. Previous work on multimodal VAEs showed that there is a trade-off between quality of generated samples and their coherence (see Sutter et al, 2021). Hence, without any quantitative results regarding the quality of samples, the assessment of a model’s performance is somehow not complete in my opinion.\n- There are multiple typos in this work (e.g. eq. 10)\n\n### Questions\n- In Sutter et al, 2020, the authors propose a similar concept as yours. Using the Jensen-Shannon-divergence, the unimodal posterior approximation are forced to be similar to the joint posterior approximation. Could you share your thoughts on the similarity between the two approaches?\n- Shouldn’t the results in Table 1 for the latent representation and the results in Table 2 (last row, 2 rows at the bottom) be the same numbers?\n- In section 3.2, what do you mean by ‘it is unlikely that the performance of inference and generation will be degraded..’? Unlikely is not a very technical term. Therefore, it is difficult to assess this statement in my opinion.\n\n### Minor Comments\n- I had difficulties finding the definition of the surrogate joint posterior. I think it is defined as Gaussian distribution as well. I could only find it for the VGON case, but not for the VAE case of the proposed method. Is this correct?\n- I would appreciate to see some qualitative results for the CelebA experiment as well.", "summary_of_the_paper": "The authors propose a new approach to the class of scalable, multimodal VAEs. They propose to use a surrogate joint posterior which, using an additional sum of KL-divergence terms, helps keeping the posterior approximation of subsets of modalities similar to the joint posterior approximation.\nThe proposed method is evaluated on two different datasets, MNIST-SVHN-Text and bimodal CelebA where the proposed method shows good results.", "summary_of_the_review": "This is an interesting approach to scalable, multimodal learning with promising results regarding the learned latent representations and coherence of generated samples. Nevertheless, quantitative results on the quality of generated samples (including random samples) are absolutely necessary for multimodal VAEs in my opinion. Therefore, I can only give a weak reject at this point.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "kzSR7jW2YhJ", "reviewer": "Reviewer_MNSw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Good:\n+ Linear increase in the number of parameters with growing number of modalities is a strong feature of the presented model\n+ The computational complexity of evaluating the loss function is substantially reduced, which plays an important role with a large number of modalities\n+ The motivation and connection to the previous work is clearly indicated, and the paper is easy to follow \n\nConcerns:\n1. Only two datasets are considered, which have 2 and 3 modalities. Since the scalability of the computational cost of SMVAE with the increasing number of modalities is one of the main motivations for the suggested method, benchmarking it on more modalities seems important. The paper follows the experiments conducted in MoPoE-VAE study, with the exception of PolyMNIST of 5 modalities. Conducting this experiment would be crucial for supporting the main claims of the paper and completing the benchmark against MoPoE-VAE.\n\n2. The evaluations for several experiments do not have the same results as reported in the MoPoE-VAE paper. this affects especially Table 2, where the results of MoPoE-VAE are often a lot worse than in the original publication(also, Table 2). As far as I can see, the results of SMVGON in table 3 all lie within the reported confidence intervalls\n\n3. The cross-generation improvement is one of the main points of the paper. However it is not really supported by the presented MNIST-SVHN-Text experiments. Out of 9 “cross-generation” experiments in Table 3, where the modality of interest is not part of the input (e.g. generating text from MNIST and SVNH), only 4 result in SMVAE outperforming MoPoE-VAE and MMVAE (as indicated by bold coloring, see minor concern below)\n\n4. Very few qualitative examples of MNIST-SVHN-Text, and none of Bimodal Celeba \n\n\nSmall concerns:\n-almost all tables miss proper description and caption. e.g., in Table 3 it is guesswork to figure out which of the header rows mean what.\n- The result tables mark several results bold as the best, even though they only differ in the third digit. It is unclear whether those differences are significant\n- Some previous multi-modal VAE models are relevant but are not part of the literature review: PoE-based SVAE model arXiv:2101.07240, which is a generalisation of MVAE and has a quadratic increase in parameters with increasing number of modalities, and VAEVAE arXiv:1912.05075 that has architectural resemblance also using the surrogate joint posterior (or a joint encoder, as introduced in JMVAE study).\n\nConcern 1 is for me the most important one and it is unclear to me why this experiment has not been conducted. Otherwise the authors should clarify the result differences in Concern 2.", "summary_of_the_paper": "SMVAE aims to solve the challenge of combining multiple modalities into a coherent latent space using VAE framework. SMVAE is based on MoPoE-VAE model, the main architectural change is replacing the MoE rule with minimising the KL-divergence between the latent space distribution generated by the PoE combined experts and the joint encoder. \n\n\n(The reviewer is sorry for the slight delay for submission as he is still on vacation and marked november third as deadline for submission. In the interest of time,the following is just a short summary of my notes)", "summary_of_the_review": "The paper includes a novel formulation of  multi-modal VAE learning, but the experimental section does not support the main claim of scalability of training and the results of the strongest reported baseline have large differences from the ones reported in the original publication. If these problems are addressed, I can see raising my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "eOl2Qphhet", "reviewer": "Reviewer_yRW1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n-------------\n- The authors do a good job of explaining the works that their paper builds upon.\n- Through the experiments, the authors are able to confirm that their objective outperforms cross-generation-based objectives.\n\nWeaknesses\n------------------\n- It is not clear why the ELBO and the KL-divergence (between the all-modality posterior and subset-modality posterior) need to be optimized simultaneously. A more intuitive approach would be to learn the all-modality posterior first by optimizing the ELBO and then optimize the KL-divergence term.\n- The objective optimized in this paper appears to be a much weaker lower bound to the log-likelihood (due to the presence of the negative KL-divergence). \n- The authors do not do a very good job of motivating the problem of learning representations when cross-modal generation is difficult. They should have conducted a few experiments where cross modal generation is difficult to ascertain that the representations learnt on these datasets by the baselines are bad.", "summary_of_the_paper": "In this paper, the authors propose a model for learning joint representations for multimodal data that allows inference when a few of the modalities are missing. Previous approaches addressed the missing modality problem by treating the inference network for each modality as an expert and combining the information across these experts. The proposed model claims to be useful in the following scenarios:\n1) When cross-generation is difficult (and hence, MoPoE-VAE fails).\n2) When the number of modalities is large.\n\nThe authors achieve this by learning a surrogate posterior conditioned on all the modalities while simultaneously optimizing the KL-Divergence between the surrogate posterior and a subset-conditioned posterior. The surrogate posterior is learnt by optimizing a variational lower bound to the log-likelihood objective.\nThe authors evaluate the representations learnt by the model on secondary tasks and report improved performance. The generated modalities also perform well on secondary tasks.\n", "summary_of_the_review": "Overall, the paper does a reasonable job of explaining the background for multimodal retrieval as well as conducting experiments to demonstrate their superior performance. However, the motivation for their work could be improved. The motivation for choosing the specific objective should also have been discussed in detail.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: PAPER: This paper proposes a method to learn joint representations from potentially missing data when (1) cross-generation may be difficult, and/or (2) with large number of modalities. This is achieved by minimizing the divergence between a surrogate joint posterior and inferences from arbitrary subsets.
DISCUSSION: The reviews and discussion brought many relevant issues and concerns. The authors submitted a revised version that improved the clarity of the paper and added an important experiment with PolyMNIST. In their responses, authors also addressed some misunderstanding about JMVAE-KL. The comparison with a relatively similar work, from Sutter et al., 2020, was only mentioned in the related work, with no direct comparisons. Also, the authors did not directly address the issue of studying tradeoffs between quality of generated samples and their coherences. It should also be noted that the advantage of the proposed SMVAE is marginal when the number of modalities increases, for the latent representation experiments on PolyMNIST.
SUMMARY: Enthusiasm for this paper was not unanimous. The reviewers brought some concerns about its differentiation with priori work, such as Sutton et al., 2020, and about a more detailed analysis of the tradeoffs. While the clarity of the paper improved during the revision, a good number of issues remained. I am leaning towards rejection.
Decision: Reject