Mitigating the Limitations of Multimodal VAEs with Coordination-Based Approach
Authors: Masahiro Suzuki
Yutaka Matsuo
Venue: Submitted to ICLR 2023 (2023)
Keywords: mutlimodal learning
deep generative models

Abstract: One of the key challenges in multimodal variational autoencoders (VAEs) is inferring a joint representation from arbitrary subsets of modalities.  The state-of-the-art approach to achieving this is to sub-sample the modality subsets and learn to generate all modalities from them. However, this sub-sampling in the mixture-based approach has been shown to degrade other important features of multimodal VAEs, such as quality of generation, and furthermore, this degradation is theoretically unavoidable. In this study, we focus on another approach to learning the joint representation by bringing unimodal inferences closer to joint inference from all modalities, which does not have the above limitation. Although there have been models that can be categorized under this approach, they were derived from different backgrounds; therefore, the relation and superiority between them were not clear. To take a unified view, we first categorize them as coordination-based multimodal VAEs and show that these can be derived from the same multimodal evidence lower bound (ELBO) and that the difference in their performance is related to whether they are more tightly lower bounded. Next, we point out that these existing coordination-based models perform poorly on cross-modal generation (or cross-coherence) because they do not learn to reconstruct modalities from unimodal inferences. Therefore, we propose a novel coordination-based model that incorporates these unimodal reconstructions, which avoids the limitations of both mixture and coordination-based models. Experiments with diverse and challenging datasets show that the proposed model mitigates the limitations in multimodal VAEs and performs well in both cross-coherence and generation quality.

Under review as a conference paper at ICLR 2023
MITIGATING THE LIMITATIONS OF MULTIMODAL VAEs WITH COORDINATION- -AANII APPROACH
Anonymous authors Paper under double-blind review
ABSTRACT
One of the key challenges in multimodal variational autoencoders (VAEs) is infer- ring a joint representation from arbitrary subsets of modalities. The state-of-the-art approach to achieving this is to sub-sample the modality subsets and learn gen- erate all modalities from them. However, this sub-sampling in the mixture-based approach has been shown degrade other important features of multimodal VAEs, such as quality of generation, and furthermore, this degradation theoretically unavoidable. In this study, we focus another approach learning the joint representation by bringing unimodal inferences closer to joint inference from all modalities, which does not have the above limitation. Although there have been models that can be categorized under this approach, they were derived from dif- ferent backgrounds; therefore, the relation and superiority between them were not clear. To take unified view, we first categorize them as coordination-bant -based multimodal VAEs and show that these can be derived from the same multimodal evidence lower bound (ELBO) and that the difference in their performance is related to whether they are more tightly lower bounded. Next, we point out that these existing coordination- based models perform poorly cross-modal genera- tion (or cross- coos--ooeee)cee because they do not learn to reconstruct modalities from unimodal inferences. Therefore, we propose novel coordination- based model that incorporates these unimodal reconstructions, which avoids the limitations of both mixture and coordination-bast datasets show that the proposed model mitigates the limitations in multimodal VAEs and performs well in both cross-coherence and generation quality.
models. Experiments with diverse and challenging
INTRODUCTION
Deep generative models have recently shown high performance multimodal data, including images and captions. In particular, multimodal learning with variational autoencoders (VAEs) (Kingma & Welling 2013, Rezende et al. (014) has attracted much attention to obtain joint representation from modalities (Wu& & Goodman 2018 Shi et al. [019 /Sttter et al. 202I). Such representations can be used for predicting common concepts from these modalities or for generating other modalities.
An essential challenge of multimodal VAEs inferring a joint representation from subsets of modalities. State-of- the-art models attempt to accomplish this by training to generate all modalities from a joint representation inferred from missing (or sub-sampled) modalities. These are called mixture-based multimodal VAFs (Daunhawer et al. 2021a), and examples include MMVAE (Shi et al. 2019) and MoPO (Sutter et al. 2021) However, it has been pointed out that the quality of modality generation is lower than that of unimodal VAES, and furthermore, pross-generation between modalities, or cross-coherence, can be degraded. This is an inherent limitation on mixture- based VAEs due to modality sub- bub-sampling during training, and theoretical and empirical evidence shows that this limitation cannot be avoided (Daunhawer et al. 2021a).
To alleviate this issue, we focus on another approach in multimodal VAEs that brings the representation inferred from each modality closer to that inferred from all modalities. This approach avoids the
Daunhawer et al. 1a) also include special case of MVAE Wu & Goodman 2018 in mixture bassd models. However, we focus the mixture-hased nature of cross goss-generation of modalities from subsets as their shortcomings, so exclude MVAE, which does not perform cross-generation.

---

Under review as a conference paper at ICLR 2023
limitation of degraded generation quality because its objective does not include generation from sub-sampled modalities. Models that fit this approach have been proposed, but their objectives are derived from different backgrounds. For example, MVTCAE (Hwang et al. 202T), which can be regarded as one of them, is derived its objective from the total correlation; therefore, its relationship to the multimodal VAEs' objective, multimodal evidence lower bound (ELBO), is not clear. Moreover, another model of this approach, MMJSD (Sutter et al. 2020). has been shown experimentally to perform worse than MVTСAE (Haag et al. 2021), ) but the theoretical reason for this is unclear.
We first categorize them as coordinaation-based multimodal VAEs 2 and show that these models can be viewed unified, i..,, all are derived from multimodal ELBO, and that the differences in models correspond to whether they are more tightly lower bounded. We also prove that MVTCAE encourages unimodal posterior to approach the average of joint posterior, which might be one of the reasons for learning good inference. Next, we point out that the existing foordination-baser VAFs still have a shorteoming in cross-coherence since no generation from unimodal inferences is included in the training. We then propose novel objective that introduces unimodal reconstruction terms. Since this objective does not include tross-generation, it also avoids the generation degradation issue of the mixture-based models while mitigating the issue of coordination- based models. Note that model alleviates the issues by only changing the objective, unlike other methods for improving multimodal VAËS by changing the architecture, such introducing additional latent variables (Tsai al. 2018 Hsu & Glass 2018 (Sutter et al. 2020 Palumbo et al. 2022).
We conducted experiments five diverse multimodal datasets, including challenging ones, e-g.. those that cannot be adequately trained with existing models or require additional architecture for adequate training. We confirmed that the proposed method outperforms existing mixture- and coordination-basee models in terms of cross-coherence and generation quality.
2 MULTIMODAL VAES
Suppose that we have a set of multimodal examples, where each example is a set of M modalities X = {x 1MM m We assume that these examples are derived from data distribution Px(X) and that each example X has a corresponding common latent concept z, i.e., joint representation.
Given a training set {X(6) /
our goals are to infer a joint representation from the subset of modalities Xs C x and to generate another subset X C X via that representation. In other words, we aim to obtain an inference p(z) X) and a joint distribution p(X, z) = IIx., Ix,xxx Ex p(x,m (z))(z).
To achieve this, it is natural choice to use VAEs (Kiggma & Welling 2013). deep generative models that can learn inference to acquire representations from data in addition to the generation. The generative model (x [z) = mI E (x,, (z) is parameterized by deep neural networks, and the prior of the latent variable is set standard Gaussian p(z) = -N(0,,
(O, I). The objective of VAEs is to maximize the expected tog-likelihood over data distribution Since it is tractable to optimize this likelihood directly, introduce an approximated posterior go X) and instead optimize the following evidence lower bound (ELBO) on the expected log-likelihood:
C(O, Φ) = (x) [x (x) x)) [log po (XIz)) DKL (z) (z| p(z))] < (Pe (I)
In this paper, the model group that maximizes this multimodal ELBO is collectively called multimodal VAEs. We aim to optimize Eq. Io obtain joint representation and generate modalities from it given modality subset Xs. However, Eq. ooly includes the inference given all modalities; therefore, does not learn to infer representations from subsets.
2.1 MIXTURE- BASED MULTIMODAL VAES
An inference from subsets is commonly expressed as combination of unimodal posteri- ors from each modality. Product of experts (PoE) (Hinton) 2002; |Wu & Goodman 2018
9 PoF Xs) p(z) [Imxxmxxxx. 9mm Xm and mixture of experts (MoE) q Mar (z) [z(xs)
TTh term coordination is taken from coordination representations, category of multimodal learning Bal trusaitis 2018), in which representations inferred from different modalities are learned close together. We denote a subset of the multimodal set X by Xs respectively, where S represents subset of (1..., M).
Tom the unimodal inference of modality Xm-
2

---

Under review as a conference paper at ICLR 2023
Emx,, EXs EX 9dm (z/xm) are known ways to aggregate inferences. Moreover, the following mixture of products of experts (MoPoE) (Sutter et al. (021) generalizes them:
MaPaB (x 46
Σ
we
II
p(z) (2)
Σ
w890s (z|Xs))
(x|Xs))
(2)
8xXgEP(X)
Xg
S:XgEP(X)
where E E ws Wg [0, and P(X) ) a powerset of X. This MoPoE aggregation is
equivalent to PoE when X5 X and MoE when Xs| for all Xs E P(X).
Substituting the MoPoo-aggreeateed joint posterior (Eq. into Eq. 1 we obtain the following:
[(0, 6) 2 Bxx(X)
Σ
we (xge - DRu PoB (2|xs(6 = (0,
(0,
(:XyEP(X)
(3)
Multimodal VAEs with Eq. are called mixture-based multimodal VAEs (Daunhawer al., 2021a)
MMVAR (Shi et al. 2019) and MoPoE-VAL (Sutter et al. 202T are members of this family.
Comparing with Eq. [ we can see that the reconstruction term (the first term in the expectation) of i designed to generate all modalities from the sub-sampled modality subset. In other words, mixture-based multimodal VAES encourage cross-genme rross-generation during training, aiming at inferring a consistent joint representation from subset modalities and generating modalities from it.
However, mixture-based VAFs have a severe issue in generating modalities, which is one of the essential goals of multimodal VAEs. Daunhawer al. (2021a) show that the following inequality holds between Eq. and the expected marginal likeliio:ooo::
(x) (loxx))oooooeX))) pe (X)) 2 A(X) + LM ())
where AXX Es.xsEP(X) W. ( (I.....rr Xx) is the discrepancy independent of the learning parameters. This inequality means that multimodal ELBO of mixture-based VAFs cannot be tighter than this discrepancy (XX) no matter how much maximized. Therefore, the quality of their generating modalities always degrades compared to unimodal VAEs. Moreover, this discrepancy, by definition, depends on how much different information one modality subset X(1....MI AfY.s has compared to the rest of them Xs. Therefore, the more information specific to each modality, the larger this quantity and the worse the generation quality.
This also leads to degrading cross-coherence (Shi al. 2019), the ability to generate the same semantic meaning across modalities conditionally. Intuitively, mixture- based VAEs are required to take sub-sampled subsets inputs generate the rest, but these inputs have insufficient information for that, resulting in degraded cross-generation.
2.2 AVOIDING THE LIMITATION OF MULTIMODAL VAES
What brings about this irreducible discrepancy A(X)? According to the proof by Daunhawer et al. (2021a) (see B.4 in Daunhawer et al. (2021a)), it is due to the inclusion in the objective of cross-generation term, i.e., a term that generates unobserved modalities from observed modalities. Therefore, one promising way to avoid this limitation is to not include in the objective of the multimodal VAE the cross-generation term
Note that MVAE (Wu & Goodman, 2018), which first uses PoE for the joint posterior, uses the sum of ELBOs over all modality subsets as the objective, but does not include cross- generation and only includes reconstruction of subsets"] thus enabling high-quality generation. However, MVAE cannot perform cross-genent cross-generation between different modalities because its objective does not include terms to encourage it. Despite this significant drawback, determining how to work around the limitation of mixture- based models leads to our proposed model described later.
H "X Mys.....yy..... (Ks) is the conditional entropy of XX1.... given Xs. p(Xss st 22]] (where X Xs), ie., modalities to be term discussed here.
'Nte that the reconstruction term (x(x) s) [logg[(XX
generated are subsets of the input, is not included in the cross-generatin
Wu & Goodman (018)) state that MVAE utilizes sub-sampling training paradigm, but this means "EI.BOs with sub-sampled modality as input", i.e., each ELBO does not include cross-generation
3

---

Under review as a conference paper at ICLR 2023
3 COORDINATION-BASED MULTIMODAL VAES
МVTСАЕ (Hwang et al. 2021 and MMJSS) (Sutter et al. 2020 have been proposed to learn to infer joint representation from subset modalities by bringing joint and unimodal posteriors closer together. We find that they omit sub-sampling in the objective unless the posterior is mixture distribution and thus can learn the joint representation without being affected by the limitation of the mixture-based VAEs. We first categorize them as unified coordination- multimodal VAES
3.1 MVTСАЕ AND MMJSD
The objectives of MVTCAE and MMISD are expressed as follows:
(2)xm.)) (MVTCAE) (5)
Epa(X) plog - EM Tm TmD Dm = Dx (yo.. (z)xm) [z)((z) (MMISD), (6)
where Σm Tm 1, km [0, 1], and function fM defines the mixture of {g (z) (z(x)))) } Im= For simplicity of notation, we use qáo (z|xo) = p(z). We denote the entire objective of MVTCAE and
MMISD (0, Φ) and L (6, (), respectively.
MVTCAE is derived as a lower bound for the total correlation When M = 2, this model is equal to JMVAE (Suzuki et al. 2016). JMVAE uses network for joint posterior separately from unimodal posteriors, while MVT AVTCAE represeesssss joint posteriors by PoF--aggregating unimodal posteriors.
MMJSD is derived by replacing the KL divergence of the multimodal ELBO with the -hasen-Shannon divergence. It is proved (in Lemma in Sutter et al. (2020)) that L (0, +) lower bound for multimodal BLBO with J'M (z) X)) replaced by the dynamic prior: p' (z X) = l'(ge... (z) xm)xm.)) )) m=))
where is function that encompasses each unimodal posterior, and [Sutter al. (2020) choose
the weighted PoE. This replacement can be thought of as introducing the prior knowledge that all
modalities have the same joint representation (Sutter et al. 2020).
Note that in Sutter et al. 2020), the posterior of MMJSD is assumed to be MoE, which is affected by the limitation of mixture- based VAEs. Since we would like focus on the properties of MMISD nature, we call MMJSD-POR P the model that replaces the posterior Eq. with PoE. Compared to the objective of MVTCAE, the direction of KI. divergence in Eq. 6 is reversed: MVTCAE uses forward KI. divergence, while MMJSD(-PoFF) uses reverse KI. divergence. Hwang et al. (222)) show that MVTCAE performs better than MMJSD by experiment.
contributing to its coordination-bass
3.2 THE RELATION BETWEEN COORDINATION -AASED MULTIMODAL VAES
Because MVTCAE and MMJSD were proposed for different purposes, their relationship is unclear. In particular, since MVTCAE was derived from the total correlation, it is unclear whether it belongs multimodal VAEs (i.e., valid ELBOs). We first show that MVTCAE multimodal ELBO.
Lemma The objective of MVTCAE C (0, Φ) is equal to the multimodal ELBO in Eq. with dynamic prior as its prior.
weighted
Proof 1 Following MMJSD, we introduce a weighted boE-d dynamic prior for L (6, ():
Epp(X) (aoo))))oottx (X)))) DD---DD-AA Dxz
(z]]
| c (0,)). .).0
= Epp((X) [locsix)llogpo Egg (x)x)
Σ Tmm log (x(2) Tm log 4om (z/xx) xmm
Therefore, MVTCAE can be considered a member of multimodal VAEs. This indicates that the ob- jective of JMVAE is also multimodal ELBO. Note that no assumption made about the approximate posterior in Lemma I and thus holds whether it is aggregated by PoE or MoE. Next, in special cases, the objective of MV TCAE an upper bound on the objective of MMJSD.
*The coordination- hased VAFs limited to two modalities are not summarized here because they are not scalable. See Section 5 for a deseription of them.
We changed the coefficients and removed non-optimized terms from the original (Hwang al.
2021
4

---

Under review as a conference paper at ICLR 2023
Lemma 2 If the posterior is MoE-Maaaas agggagggre,,,ed the objective of MVTCAE (0, Φ) is an upper bound on that of MMJSD L (), L.e., the following inequality holds: L (0, 6) 2 _ (0, o).
Proof 2 Lemma I shows that the objective of MVTCAE is equal to a multimodal ELBO with dynamic prior. In addition, Lemma 2 in Sutter et al. (2020) states that the objective of MMJSD is lower bound on multimodal ELBO with dynamic prior and MoE-aggregatee posterior. Therefore, the objective of MMJSD a lower bound on that of MVTCAE if the posterior MoE- agge-aggge D
Lemma 2 indicates that MVTCAE better approximates the true log- logllikelihood and thus promotes the good generation, which provides one possible explanation as to why MVTCAE performs better than MMJSD. Note that Lemma 2 comes from the fact that the joint posterior of the KL divergence in MMJSD assumes MoE; therefore, it is still unclear whether this holds when the posterior is PoE.
Furthermore, show why this direction of KL divergence for MVTCAE is good for multimodal VAES from a different perspective.
Lemma 3 The KL divergence for each (Eq. brings the average of the joint approximate posterior 9o (z(z) Xm) = (z(2 (P(cC)PP(XX//))) Xm d closer to the unimodal posterior qom Xm))
Proof 3 Eq. is transformed as follows 10
Px)( (nxx()s(x)) (x]xm))) (x(Cemmiie + FPddxm)
(x]xm))]] (7)
where (m(///mm2xx,n is conditional mutual information and holds greater than O. Therefore, maximizing the negative KL term (Eq. 5 leads to minimizing the second term in Eq. 7
Lemma holds for any approximate posterior. Therefore, another possible reason MVTCAE is good as multimodal VAEs is that maximizing Eq. exxplicitly brings the unimodal posterior closer to the joint posterior with the unimodal input.
the joint inference is PoE-aggregation with Gaussian, the approximate posterior of any input can be computed analytically. Therefore, the second term in Eq. can be to zero by using PoE-aggregate posterior go (2|xm) p(z) [z] Xm) instead of unimodal posterior 2s... (z Xm). From now on, we use the Pggreggee posterior.
COORDINATED AND UNIMODAL RECONSTRUCTED MULTIMODAL VAE
If the inference can be properly learned, the inference from arbitrary set of input Xs can be com- puted analytically as qo (z) x)) = p(z)) Imxx.,, To... (z) xm) due to the nature of PoE. However, c (0, Φ) only includes reconstruction from all modalities and does not include generation from arbitrary modalities; thus, the performance of modal generation might be degraded. On the other hand, cross -rose-gener in the objective conversely promotes generation degradation due to the limitation of mixture-based models.
We propose to include the term for unimodal reconstruction in the objective to experience the gener- ation from unimodal posterior during learning while avoiding these dilemmas. The reconstruction term in Eq. can be rewritten as
(x)(X) (loggoo = Bpa(x)
1
1 Σ Epa(Mm)
[B (a (log poog po po
(8)
where all inferences in Eq. are made using Pol-aggreeated posterior.
This rewritten objective can be converted to include all all-modality and unimodal reconstruction, which mitigates the shortcoming of coordination-base VAEs while avoiding the limitation of basedd
VAEs. We call this Coordinated and unimodal Reconstructed Multimodal VAE (CRMVAE).
can also be transformed into The expected value for x., of the sum of the second term in Eq. 8 reconstruction that includes another input as follows:
(x)x... Epa (Xm) (z) [log Po xmm [2]]
Epp(Xn. X,n))
E (xlxxxX x..) [log po (X x)]]
10This is application of the derivation used for one two modality inputs in Hoffman & Johnson 2016
Vedantam al. 2018) multimodal inputs.
5

---

Under review as a conference paper at ICLR 2023
Therefore, optimizing the unimodal reconstruction for each modality leads to the possibility of reconstruction from any modality. This performance can be generalized beyond reconstruction to improve the performance of cross-modal generation.
Note that thcoretically, the same is true for the left-hand side of Eq. S i.c., from the nature of PoE. should be possible to learn inferences from any modality simply by optimizing the left-hand side of Eq. B However, some inferences might be ignored in actual optimization. In other words, it is true that if each inference adequately learned, the joint inference will be adequate, but the converse is not necessarily true. Our proposal is empirical and attempts to alleviate this problem of being ignored by having each modality experience inference and generation. This simple method is effective for cross-modal generation in coordination-basee setting, as we will show later in the results.
In the experiment, we set the coefficient in Eq. 8 from 1 to mm prevent the expected value term from increasing when the modality increases and having too large an effect on the overall objective.
5 RELATED WORKS
There have been many studies of multimodal VAEs (Suzuki & Matsuo, 2022), and the early ones before mixture- based required additional inference models. (Szzki et al. (2016) proposed ap- proximate the unimodal inference models to explicitly approach the joint posterior of VAEs, called JMVAE. Vedantam et al. (2018) introduced TELBO, where the objective is composed of the sum of ELBOs that take each set of modalities as imput. V (Korthals et al. 2019) can be regarded as a combination of JMVAE and TELBO, and VAEVAE (Wu & Goodman 2019) excludes KL divergence term between the joint inference and prior from VA Except for TELBO, they can be regarded as precursors to the coordination-bon based approach; however, the number of inference models needed for these models grows exponentially with the number of modalities.
One important research direction in multimodal VAEs is the introduction of additional latent variables.
which includes hierarchical latent variables (Sutter & Vogt 2021 Wolff et al. 2021] Vasco et al. 2022) and modality-specifft latent variables (Tsai et al. 2018 Hsu & Glass| 2018 Sutter et al.) 2020
Lee & Pavlovic 2020; Daunhawer et al. lb; Palumbo et al. (022). This paper aims to improve the performance of multimodal VAES in a different direction from them, i.c., whether limitations of existing mixture-basse based models can be mitigated by changing the objective while maintaining the same architecture. Note that our proposed model and the introduction of additional latent variables orthogonal concepts and can be easily incorporated into the proposed model.
EXPERIMENTS
We use the following five diverse datasets: MNIST-SVVNN-Tex (Sutter et al., 2020), PolyM- NIST (Sutter et al., 2021). Translated-Poly PolYMNIST (Daunhawer et al. 2021a), CUB Wah et al. (2011). and Bimodal CelebA (Sutter et al. 2020). For more description of the dataset, see the appendix. These are commonly used in studies of multimodal VAEs, but to our knowledge, this is the first study in that experiments have been conducted on all of these datasets. In particular, CUB and Translated-PolyMNNIS
were noted to be challenging datasets that could not be successfully trained in existing models (Daunhawer al. 2021a). It also shows that Bimodal CelebA and CUB could only be learned in models with additional latent variables (Sutter et al. 2020 Palumbo et al. (022).
For models trained in all modalities, we evaluate both 1) cross-coherence and 2) generation quality given arbitrary modalities. We measure each of them in terms of 1) the test accuracy of a pre-trained classifie that predicts labels from cross- cross-generss geseeatedd images and 2) the Fréchet inception distance (FID) (Heusel et al. 2017) in cross ross-generated modalities. For MNIST-SVHN-Tee. joint coherence also verified by generating random images in all modalities and evaluating whether all classes predicted by them are consistent. Note that we evaluated the generated results of the text modality in MNIST-SVHN. -ext in terms of the Levenshtein distance from the true text (without space characters)
since we found that the pre-trained classifier from Sutter et al. (2021) did not correctly evaluate the generated results (we discuss in the experimental results.) To assess the cross-coherence for CUB, we
1ihhs leads to change the balance between the reconstruction and KL. terms compared to other models as the number of modalities increases. This will be examined in Appendix H
We used the pre-trained classifier in PolyMNIST and MNIST-SVHN-TTet from Sutter al. 2021

---

Under review as a conference paper at ICLR 2023
Table 1: Cross-coherence on MNIST-SVHN-Teas M, S, and T represent MNIST, SVHN, and Text. The top row shows the modalities generated, and the second row shows modalities conditioned when generating them. For example, the second column of the MVAE results (accuracy 0.30) represents the cross-generation from S to M. A+B means that the modality in the row above is generated from the modalities A and B. R represents the joint coherence of all modalities in random generation.
M
T
MODEL
R
S
S+T
M
T
M+T
M
S
M+S
MVAE
0.04
0.30
0.11
0.30
0.36
0.29
0.67
0.13
0.20
0.22
MMVAE
0.12
0.81
0.99
0.91
0.39
0.44
0.41
0.48
0.40
0.44
Mixture
MoPoE- VAF
0.12
0.82
0.99
0.94
0.35
0.37
0.36
0.49
0.42
0.51
MMJSD
0.09
0.81
0.99
0.90
0.39
0.41
0.40
0.44
0.37
0,40
MMJSD-POR
0.15
0.12
0.78
0,80
0.55
0.73
0.86
0.71
0.14
0.74
Condination
MVTCAE
0.09
0.23
0.86
0.90
0.59
0.78
0.80
0.76
0.27
0.79
CRMVAE
0.15
0.60
0.97
0.97
0.73
0.87
0.87
0.91
0.71
0.95
Table 2: FID scores for cross-
SVIIN
MNIST
SVII
generation. Lower is better.
0
4
42
O y
2
4 9
0
4 2
42
MODEL S S->M
M->S
MNIST
SVIN
Text
MVAE
39.0
36.0
MMVAE
91.8
307.9
MVAE
3
7
0
91
MoPoE- VAE
85.3
312.8
MoPoE
0 4
2 7
MMJSD
86.7
287.0
MVTСAЕ
0
8
9
0
1
AMJED-POE
70.0
141.4
CRMVAE
0
H
2
4
7
9
2
MVTСAЕ
41.6
31.8
CRMVAE 24.1
37.3
Figure 1: Visualization of cross-generation.
use the cosine similarity between the latent space of the two inferred modalities since there labels. We also used this evaluation for Bimodal CelebA
For the network architectures, we follow Sutter al. (2020): Daunhawer al. (021a). with 64- dimensional latent variables for Bimodal CelebA and CUB, 32 for MNIST-SVHN. -exxt and 512 for PolyMNIST and
The distributions for image modalities are set to Laplace, and text is set to categorical. We use Adam (Kingma & Bal 2014) for optimization with a learning rate of 0.0005 and a mini-batch size of 256. We set 3 (the coefficient of the KL divergence term in the multimodal ELBO) to 1 for PolyMNIST and MNIST-SVHN-TT Tm to M+ and Ws to 2 We train 100 epochs for UNIST-SVHN' Text, 150 for CUB, 200 for Bimodal CelebA, and 500 for PolyMNIST and Translated- Paliattdd-oolyyMaMa PONSM. All experimental results the average of three runs with different seeds. We implement them using Py PyTorch (Paszke et al. 2019). Our experiments were performed using two machines with five NVIDIA Titan RTX GPUs. We compare CRMVAE with MVAE, mixture-based VAEs (MMVAE, MoPoE, and MMISD), and coordination-ba based VAEs (MVTCAE and MMJSD-PoE), all with the same architecture settings.
MoaislatedPPllMNNISS
and 0.1 for Translated-PolyMNNSS
6.1 MNIST-SVHN-TEN
Table 1 shows the joint and cross-coherence results, and Table 2 shows the FID scores of images MNIST-SVHN-Tex.. First, we can confirm that MVAE has high generation quality but the lowest cross-coherence since it learns only the modality reconstruction. We next find that cexture-based models perform poorly in cross gonss-generation of SVHN and Tex According to the qualitative results shown in Figure [ the cross- gross-generation of Text MoPoE- VAE is unsuccessful. While Table [ shows that cross- gonss-eenerrtion of MNIST appears to perform well, the qualitative results show that the generated images are average with variation. Table 2 indicates that FID scores of mixeure-based models on MNIST are significantly lower than these of MVAE, which corresponds to these results.
Existing coordination-bact based models mitigate this limitation and perform better on SVHN. Further- more, FID scores show that the coordination based models terms of generating quality and are comparable MVAE. Comparing MMJSD-PoE and MVTCAE, MVTCAE performs better in terms of both cross cross- cooss-co and generation qual-
models significantly outperform the mixture-
Note that the results of Text differ significantly from the results of previous studies because the Levenshtein distance is used for the generated results, not for the pre-trained classifiers. We find that the evaluation using the I Levenshtein distance consistent with the qualitative results.
7

---

Under review as a conference paper at ICLR 2023
PolyMNIST
Translated PolyMNIST
Clossiiiation accuracy
Classifiiation accurscy
F10 500
FID
Number of input moda ities
Number input modelities
Number of inpul modelities
Number of input modaitiis
Figure 2: Performance evaluation on PolyMNIST and Poa MNIST. The left panel for each dataset shows the cross-coherence given subset of size corresponding each number and the right panel shows the FID score given subset (lower better). The markers are the means the three runs, and the error bands are their standard deviations.
ity, confirming the effectiveness of MVTCAE experimentally. However, the performance of the coordination-based is degraded, especially in cross-generation from SVHN.
CRMVAE mitigates coordination- based issues and significantly improves cross-coherence perfor- mance. In particular, CRMVAE outperforms all existing methods, including mixture-based methods, for cross-coherence of SVHN and Text. We also confirm that CRMVAE retains the advantage of the coordination- based models, i.c., high-generation quality. In particular, we have confirmed the highest quality of cross-genen goss-gennrattion on MNIST among all models that include MVAE. The visualization results also confirm that CRMVAE achieves high coherence while maintaining high quality.
The performance of the joint coherence results with random images is poor compared to other studies, which might be due in part to the different methods for evaluating text. The performance of the coordination-based method relatively high but not significant. The relationship between joint coherence and coordination-basae should be studied further.
6.2 POLYMNIST AND
Figure 2 shows the results on PolyMNIST and Translated- PollMMtIS. Based on the above results, we select HoPoE-VAE: and MVTСAE as mixture-based and noordination-basee models for comparison. First the results of the existing models on PolyMNIST show the same trend as on MNIST-SVHI -ext, ie.. there is a trade-off between cross-coherence and generation quality. CRMVAE mitigates these limitations and achieves the highest cross-coherence while maintaining high generation quality. Note that it might seem strange that both MVTCAE and CRMVAE produce lower quality when the number of modalities increases. This is thought to be due to a decrease in the diversity of the generated data distribution the number of modalities increases.
Translated- Paae--PoyMMSSSSS is a slightly modified PolyMNIST, but Daunhawer et al. 2021a) point out that the generation quality and cross-coherence performance of mixed-based models is significantly degraded on this dataset. Our experiments also confirm that most existing models fail to cross- goos--geeeete across modalities. CRMVAE, on the other hand, clearly improves cros-ccoheeence. although not as much as PolyMNIST. Next, CRMVAE has the highest generation quality, significantly outperforming the mixture-based models. See Appendix for visualization of the generated images.
6.3 CUB AND BIMODAL. CELERA
Figures 3 and show the quantitative and qualitative results for CUB and Bimodal CelebA. We examine generation quality and cross-coherence performance as an ablation, when varying First, the FID score indicates that MVAES have high generation quality, while MoPoE-VAES have significantly lower quality. In most cases, the FID score does not change much or worsen when is increased, while the FID score for MoPoE- -VE improves slightly. This is thought to be because increasing 3 in MoPoE- -VE corresponds to making the distribution closer to the standard Gaussian, which diminishes the effect of generation from sub-sampling. Furthermore, the performance of the coordination- based methods, including CRMVAE, is close to that of MVAE, indicating that they do not have generation quality issue these challenging datasets.
8

---

Under review as a conference paper at ICLR 2023
6125
1: black
distinct
and rounded
2: the wings are brown. long and narrow and have white markings on the
seconsaries have bird be soems to hare a long dark calored bill with wiste ring around
light brown.
above the water
color
299
0.73
240
NaTie-WA
MYKE
S: bidd with simost
wingspan
made catching
6.033
229
CRMMAE (ours)
NVVE
MoPoE
SIVTCCEE
CRNVAE
6425
Figure 3: Left: Performance of the cosine similarity of the two modalities in latent space (higher is better, left) and the FID score of the image generated from text (right) for varying 3 in the CUB dataset. Right: Visualization of the CUB images generated by each model (3 = 10, bottom) conditioned on each text (top). The number in each column of each model represents the number of the text being conditioned. Different rows correspond to different sample results.
DastPoatton Statt maaee)
8.50
Crere berkett
180
0.00
8.00
2.45
160
ManuE
VOTCAE
MoPoE
MYTCAE
CANVAE
0.00
6.00
0:26
160
6.3.
5.6
10:2
8
5
Figure 4: Left: Performance of the cosine similarity (left), classification accuracy (center), and FID score (right) for varying 3 in the Bimodal CelebA dataset. Right: Visualization of the Bimodal CelebA images generated by each model (3 = 5, bottom) conditioned on each text (top).
Next, the evaluation of cosine similarity shows that MVAE and MoPoE-VAE do not change that much when / is increased, while the coordination-base based models, including CRMVAE, improve when 3 is increased. This is because, in the coordination- based models, the term adjusted by 3 corresponds to the term that brings the inference of all modalities closer to that of each modality. Compared to MVTCAF, CRMVAE performs better for Bimodal CelebA for all values of / and for CUB when 3 is large. Furthermore, the cross-coherence with pre-trained classifiers in Bimodal CelebA also shows that CRMVAE performs better than MVTCAE. Note that MoPoE-VAL has the highest cross-coherence value, but as the qualitative results show, this only improves classification accuracy at the expense of image quality, e.g., diversity (as is the case for MNIST in MNIST-SVHN- -ext)..).
Qualitative results showed the following limitations: MVAE failed to produce images corresponding to the text, and MoPoE-VAL produced only average or blurred images. Comparing MVTCAE and CRMVAE, CRMVAE produces more appropriately conditioned images. For example, in Bimodal CelebA, Text 3 contains a male attribute and CR produces a corresponding face image, while MVTCAE produces some female face images.
7 CONCLUSION AND DISCUSSION
We focused on coordination-batt multimodal VAES avoid the limitations of multimodal VAEs. We first discuss the new relation among coordination based multimodal VAES, point out the issue of unimodal inference not learning from reconstruction them. and then introduce a new objective involving unimodal reconstruction. Experiments show that the proposed model mitigates exist- ing models' limitations (both mixture-based and coordination- based) and shows overall superior performance on diverse datasets for generation quality and cross-coherence.
One limitation of CRMVAE is that there still remains a trade-off between generation quality and cross-coherence. For example, in the PolyMNIST experiment, CRMVAE performed best in cross- coherence, but the FID score was worse than MVTCAE. The performance of joint coherence was also confirmed to be limited. In addition, although we used several diverse datasets to evaluate performance, these datasets are much smaller than the various multimodal information obtained from the real world. Therefore, should validate our model using such large multimodal information.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency.
Multimodal machine learning: A survey and taxonomy. IEEE transactions pattern analysis and machine intelligence, (2): 423-443, 2018.
Imant Daunhawer, Thomas M Sutter, Kieran Chin-Cheong, Emanuele Palumbo, and Julia E Vogt. On the limitations of multimodal vaes. arXiv preprint 1r.00:211000412.. 2021a.
Imant Daunhawer, Thomas M Sutter, Ričards Marcinkevičs, and Julia E Vogt. Self-supervised disentanglement of moddalit-sspecifff and shared factors improves multimodal generative models. Puttern Recognition, 12544:459, 2021b.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1711-180)) 2002.
Matthew D Hoffmman and Matthew J Johnson. Elbo surgery: Yet another way to carve up the evidence lower bound. In Proc. Workshop Adv. Approx. Bayesian Inference, pp. 2, 2016.
Wei-Ning Hsu and James Glass. Disentangling by partitioning: A representation learning framework for multimodal sensory data. arXiv preprint arXiv: /224, 2018
HyeongJoo Hwang, Geon-Hyeong Kim, Seunghoon Hong, and Kee-Eung Kim. Multi- view representation learning via total correlation objective. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neu- ral Information Processing Systems, volume 34, pp. 12194-12207. Curran Asso-
ciates, Inc., 2021. URL hptppes neurips. cc/paaprr/222//111 pdd
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.698) 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114 2013.
arXiv preprint
Timo Korthals, Daniel Rudolph, Jürgen Leitner, Marc Hesse, and Ulrich Rückert. Multi-modal generative models for learning epistemic active sensing. In 2019 International Conference on Robotics and Automation (ICRA), pp. 3319-3325, IEEE, 2019.
Mihee Lee and Vladimir Pavlovic. Private-shared disentangled multimodal vae for learning of hybrid latent representations. urXiv preprint arXiv:2012.1 13024, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Emanuele Palumbo, Imant Daunhawer, and Julia E Vogt. MMVAE+: Enhancing the generative
quality of multimodal VAEs without compromises. In ICLR Workshop Deep Generative Models for Highly Structured Data, 2022. URL htttps://openreviee.co net./ forum? d
B42UJTVdD25!
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bur E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conperaeetninnccccoennncncc on machine learning. pp. 1278-1286. PMLR, 2014.
10

---

Under review as a conference paper at ICLR 2023
Yuge Shi, Narayanaswamy Siddharth, Brooks Paige, and Philip Torr. Variational mixture- cexturr--ffexxe -faee-ff-eppeee expeets autoencoders for multi-modal deep generative models. In Advances in Neural Information Pro- cessing Systems, pp. 15718-15729, 2019.
Thomas M Sutter and Julia E Vogt. Multimodal relational vae. 2021.
Thomas M Sutter, Imant Daunhawer, and Julia E Vogt. Multimodal generative learning utilizing consen-shammoo-ddieegeeee arXiv preprint arXiv:2006. 08242, 2020.
Thomas M Sutter, Imant Daunhawer, and Julia Vogt. Generalized multimodal elbo. arXiv preprint arXiv:2105.02470, 2021.
Masahiro Suzuki and Yutaka Matsuo. A survey of multimodal deep generative models. Advanced Roboticx, 36(5-6):261-278, 2022.
Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Joint multimodal learning with deep generative models. arXiv preprint arXiv:1611.01891, 2016.
Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Phillppe Morency, and Ruslan Salakhutdi- nov. Learning factorized multimodal representations. arXiv preprint arXiv: 067666 2018.
Miguel Vasco, Hang Yin, Francisco S Melo, and Ana Paiva. Leveraging hierarchy in multimodal generative models for effective cross-modality inference. Neural Networks, 146:238-255, 2022.
Ramakrishna Vedantam, Ian Fischer, Jonathan Huang, and Kevin Murphy. Generative models of
visually grounded imagination. In International Conference on Learning Representations, 2018.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Jannik Wolff, Rahul G Krishman, Lukas Ruff, Jan Nikolas Morshuis, Tassilo Klein, Shinichi Nakajima, and Moin Nabi. Hierarchical multimodal variational autoencoders. 2021.
Mike Wu and Noah Goodman. Multimodal generative models for scalable weakly -upervssd learning, In Advances in Neural Information Processing Systems, pp. 5575-5585, 2018.
Mike Wu and Noah Goodman. Multimodal generative models for compositional representation learning, arXiv preprint 1rX1v:1912.05075, 2019.
A DATASET
MNIST-SVVNN-Tes (Sutter et al. 2020) is the MNIST-SVHN (Shi et al. 2019) with the addition of the text modality, which represents the English name each number as text and has diversity. Different modalities of the same example have the same digits.
PolyMNIST (Sutter et al. 2021 consists of a set of five different MNIST images with different backgrounds and handwriting styles, each of which is considered different modality.
Translated-Pe Poattedd-PlyMMMISSS (Daunhawer et al. 2021a) is composed of the same backgrounds as PolyM- NIST with the numbers downscaled and randomly transformed into position in the images. This is intended to allow for more significant variation among modalities than PNIST.
Caltech Birds (CUB) Wah et al. (2011) is dataset with bird images and their captions. We follow Daunhawer et al. a02 T.e., we treat as 64 x 64 real images without feature extraction.
Bimodal CelebA Sutter et al. 2020) is dataset of face images from the CelebA (Liu al. 2015) with additional text describing the corresponding attribute labels and changing the starting index to give diversity.
In setting of this study, the handwritten numbers were downscaled to 70% of their original size.

---

Under review as a conference paper at ICLR 2023
B DETAILED DERIVATION OF PROOF 1
(e po b)] D [x]])(( (z|x))]]]
Bxp(X) (Lg, (X 2)] - g flog go go (z|X) - -logp' [I]
M
(x)
Boupe (x [lg 90 (z|X)
Σ
um log qpm (z) Xmm)
m=0
: (
(z|x)
Σ Tm log aohn (z(xm)
M Σ m
M5
= (x)(
Poocexx//oopooXXP0 (x(())
Tm log g (x(x)
Σ
Tm
log Tom (z/xx))
m
M
log (z(x))
Σ
Tm log 90 (z|X) and >
Tm
=
m=00 0
M
=x)
(og
Σ m=0
Tm (x)x) (z/gg(z(()) log qom (z)xmm)
M
(X)
(ootexx)(oogp((((1))
umm DEL (g0(2|X)) (2 (2/xxm))
_ (0, ())
C DETAILED DERIVATION OF PROOF
(x)( [Dx (xxlggzz//X)lle (z(xm)))
/ (Plz))))a(((A)) (xm) log
yo (z)
pa(xm)
dX c/m mddxx
2am. [2) (2|xmn)
pd(xm))
/
log
9 Z X )p (ipp(X(er)
(XC/m /mmll (2m)
d. c//m dzdxm
qom xm ) X/cm Xm)
2 (z) X )p X c/ Xm)
(p((xm))
(A(z))))s(( (mm) log
iXc/mdadx,
go xm )pm (al Xmm)
+ / pa(xm)) /
2 (z) xmm
(z(z|X (x (X xm Jd c/mm log
dzdxm
Tem (z) (m))m)
/
(2,2, Nc/m xm)
pa(xm) go(z, Xc/m log ( X (aa (Xpca Xm)
dXCc/m dzdxx
/ pa(xm)
/
90 (o(z)(m)) xm) log Uo (z) xm)
dzdxm.
qo... (z xm) [Dx (ao Xm Ila... (x)xm))]]
Ts (X C/mi 2)xm) + E (pp(xm))
12

---

Under review as a conference paper at ICLR 2023
D DETAILED DERIVATION OF EQ. 7
Epa(X) (1 po 2]]]
=B)((X)
B (latelx)(logpoXX|
Bg (x|x [log po (1x
M
- -x)a(X)
2 Ean [lo
+ E (x(x))
log pa 8mm| 2)
2
m=1
M
1
(X)
) [logpo
+ 2 (o(z|X)pa(X))
log (xm 2)dzd
m=1
1 Σ
xx)((X)
2 (x)()) [log (X
90 log po (0mlz))dda z)dzdX
Restelx) (o
90 (o(2\X)pp(XXx) Xm Xm)Pa(Xm) log (x 12)
m=1
M
= -Epa(X)
Beslelx) [oogooX
+ 2 m=1
q0 (xlz\X)p((Xc/mm (dl log Polgpo(C (xm (ommlz)eee 12)
2 [xll))) [X
Σ
4o qa(z/xm)pa(xxm log (xm dzddxm.
2 m=1
gos //X// (2(2)xx))
M
2 Bpa(t) (logge
E (xp( (x llogg (xm |2]]
E DETAILED DERIVATION OF EQ.
Epp(x..))
[ggo(x) [log pe [xm] [z]]]
16 (z) (x) X) log po (xm 22d
/J 1o (z|x {1p( X0/m.m/ Xm, n dX C mi xmv) log (xm (2 dz)x dxmm
/ do (z/xm Xmm )a (Xms Xym) log Po (mm [z) d'z dx
Epa(Xm) xmn!) [Bae (2t bem xx,. [log Po x [2)])
F RESULTS ON MNIST-SVHN-TEXT WITH STANDARD DEVIATIONS
Table 3able Table Dare Table I and Table 2 with their standard deviations three trials explicitly added. The meaning of MVTCAE (rescale) explained in Appendix H
G PERFORMANCE ON THE LATENT REPRESENTATION
Table shows that coordination-based performs better input modality increases. This is a feature of the coordination- based method, which learns to approximate the joint posterior, so the amount of information to be inferred increases the number of modalities increases. On the other hand, the mixture-based method performs better given a single modality. The proposed method improves unimodal inference the most among the coordination-hase methods.
13

---

Under review as a conference paper at ICLR 2023
Table 3: Cross-coherence on MNIST-SVHNNText with standard deviations.
MODEL
R
T
S+T
M
T
M+T
MVAE
0.04 0.01
0.29 004
0.11 0:00
0.30= 0004
0.36 0.88
0..99= 0.08
0.67= 0.05
MMVAE
0.12 0.11
1.81=000
0.99
0.91 0.00
0.39 0.01
0..44=
0.41
MoPoE- VAB
0.12 0.01
0.82 0.00
0.99 0.00
0.94 oo1
0.35 0.02
0.37= 0:02
0.36= 0:02
MMJSD
0.09 0.00
0.81 0.0
0,99 0.00
0.90 0.0
0.39 0.01
0.41 0.01
0.40
AMJSD-POE
0.15 0.03
0.12 0.01
0.78 0.03
0.80 o.3
0.56# 0.1
0.73= 0.03
0.86
МVТСАЕ
0.09 +: 0.00
0.23 +: oot
0.87 0002
0.90+ 0.1
0.59 0.01
0.78 0.00
0.80 0.01
МVTСАЕ (RESCALE)
0. 1. 0:00
0.29
0.89 0.01
0.92 0.01
0.70= 0.01
0.79 0:01
0.82
CRMVAE
0.15 0.01
0.6
0.97 0.01
0.97 0.01
0.73
0.87 0:01
0.87 0.01
T
MODEL
M+S
MVAE
0.13+ + 0.01
0.20+
0.22+ oue
MMVAE
0.48 0.05
0.40+ a04
0.44
MoPoE-VaL VAE
0.49
0.42
0.5
MMJSD
0.44:t 0.02
0.37 0.01
0.40 0.02
MMJSD-POR
0.71 0.02
0.14
0.74 0.04
MVTCAE
0,76 0.02
0.27 0.02
0.79 0.02
MVTCAE (RESCALE)
0.87 0.01
0.34- 0.02
0.89 0u2
CRMVAE
0.91 + 0.02
0.7 0.00
0.95+
Table FID score of the generated SVHN images with standard deviations.
MODEL
S-*
M->
MVAE
39.0
36.0.
MMVAE
91.8+ 1.46
307 9 0.2
MoPoE- VAE
85.3 2.51
312.8
MMISD
86.7 1.94
287.0c 137.0+ 1.15
MMJSD-POE
70.0 1.17
141.4t 3.20
MVTCAE
41,6+
31.84 0.1
MVTCAE (RESCALE)
32.9
033
CRMVAE
24.1
3753
The right panels of Figure and Figure 6 show the inference performance of PolyMNIST and Translated-PolyMNiIT.. respectively. It can be seen that the proposed method performs well when the number of modalities is small, while other models perform better when the number of modalities increases. There are various possible reasons for this, but the scaling of the reconstruction error term, which will be discussed in the Appendix might be the cause.
Table 5: Performance the latent representation in MNIST-SVHN-Tex.
MODEL
M
M+S
M+T
S+T
M+S+T
MVAE
0.73 0.8
0.44 o.2
0.30 004
0.84 007
0.97 0.00
0.58 0.08
0.97
MMVAE
0,97 0.01
0.82 0.08
0.99 0.00
0.894 0.04
0.98. 0.00
0.91 0:01
0.93 0.05
MoPoE-VAL VAE
0.95
0.81 0.4
0.99+
0.96+
0.98 0.00
0.93=
0.98
MMJSD
0.97+ 0.01
0.82+ 0.02
0.99 0.00
0.89+ 0:11
0.98 0.00
0.91 0:01
0.93 0:01
MMISD-POR POE
0.75 0.01
0.14 0.02
0,94 out
0.77+ 0.22
0.99 0.03
0.95 0.05
0.99 0:03
МVTСАE
0,91 + 0.01
0.31 +
0,99 0.01
0,93 0.2
0.99 0.0
0.99 0.00
0.99
MVTСAE (RESCALE)
0.88 0.01
0.35 0.0L
0,95 0.02
0,90 0.02
0.96 0.01
0.96 0.01
0.96 0.02
CRMVAR E
0.95 0.01
0.74 o.22.
0.99 0.1
0.97 0.00
0.99 0.01
0.99 0.1
0.99
14

---

Under review as a conference paper at ICLR 2023
PolyMNIST
1.2 140
Classification accuracy
Latent representation
0.3
0.
FID score
Hod UNitid cevaao
0.5
0.4
Number of input modalities
Number input modalities
Number input modalities
Figure 5: Performance evaluation on PolyMNIST. The left panel shows the performance of cross-
coherence. The right is the performance of linear classification on the latent space given an arbitrary modality set.
Translated-PolyMNIIT
Classification accuracy
Latent representation
0.28
146
0.6
64
0.26
6.3
0.2
50
Number of input modalities
Number of input modalities
Number of input modalities
Figure 6: Performance evaluation on Translated Poliltted-PoMMMNIST...
H CONSIDERATION OF THE COEFFICIENTS OF THE RECONSTRUCTION TERM
We chose instead of the coefficient in Eq. However, this makes the reconstruction term smaller than the other models by factor of 2. To check this effect, we scaled the MVTCAE reconstruction error term by multiplying by z. Since this effect large when the modality is three or more, we experimented with MNIST-SVHN-Te.. PolyMNIST, and Translated-PPlyMMNiSS
The results denoted "MVTCAE (rescale)" the MVTСAE results aligned in scale with the reconstruction eтror term in CRMVAE. First, for ANIST-SVHN-TTeE the performances of cross- coherence and FID are slightly better with scale, but CRMVAE still better. The joint coherence was comparable to that of CRMVAF. However, the performance of the inferred representation is lower with scaling. This is because the coordination- based learns inference by reconstruction, and when this term is small in the overall expression, it fails to learn inference. Next, PolyMNIST and Translated- Pooiecdd-PPMMNSSTT improve cross-cote cross-coherence performance, but as with MNIST-SVHN -ext,, inference performance is poor. Compared to MVTCAE with rescaling, the performance of CRMVAE better, and this comparison, which the reconstruction error terms are scaled to each other, might better evaluate the effect of unimodal reconstruction included in CRMVAE.
These results show that adjusting for scale does not change the superiority of CRMVAE, other words, the higher performance of CRMVAE is not due to scaling alone.
15

---

Under review as a conference paper at ICLR 2023
MO
7
5
2
&
2
M1
7
7
J
0
3
2
0
3
2
2
5
2
0
3
8
2
M4
2
P
M1,2,3,4
20,233,4-M
4
3
3
MVAR
7
3
6
9
McRGEAEE
2
3
S
2
7
7
2
3
O
8
2
A
7
MVTCAR
2
5
0
3
3
CRMIAE
3
5
2
O
3
8
2
7
7
2
3
5
2
O
3
8
2
M0,1,3,4 -M2
M0,1,2,4-W0 M3
7
5
2
3
8
2
2
3
5
3
0
3
0
2
McROFAAE
3
5
2
McRGEAAE
7
2
5
2
0
3
8
2
3
6
2
3
a
7
7
2
3
5
2
0
3
8
2
ORMVAR
7
2
5
2
O
CRMVAF
7
7
2
3
5
2
O
3
8
2
M0,12,3 M44
MWAE
3
5
McPsBAVAE
7
5
MVTCCEE
7
7
2
3
5
2
2
3
8
2
7
7
2
3
5
O
3
2
Figure 7: Cross Cross-generation of PolyMMISS. The top panel shows examples of modalities MO to M4 (we refer to the five different modalities as MO M4), and the bottom panels show the generation of modality conditioned remaining modalities by each model.
I CONDITIONAL GENERATION ON POLYMNIST AND
Figure 7 and Figure visualize the conditionally generated images the PolyMNIST and Translated- PolyMNIST datasets.
16

---

Under review as a conference paper at ICLR 2023
M0
9
A
O
M2
7
9
M5
2
Q
7
M1,2,3,4
M0,2,3,4 MM
MVAB
O
McRsEMWE
MoRt VAG
MVTCAE
3
_
CRAVAE
CRMVAE
&
0
f
M0,1,3,4- M2
M0,1,2,4 M3
MYAB
MVAR
MoRsEAALE NALE
McRaEAAR
MVTCAE
0
MVTCAB
9
CRMVAE
M0,12233M4
MYWE
MVTCAR
CRMVAE
E
Figure 8: coos-genneaation of Tralssated-PPlyMMMIST..
17


Reviews:
[{"review_id": "zIJtdir1vi", "reviewer": "Reviewer_venE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes recent development in Multimodal VAE in extensive details. \nSpecifically, versions of the ELBO where the latent models , based on subset of modalities, are forced to be similar to\nthe latent model that is estimated with all the modalities is explained as Product of Experts like model. The paper goes on \nto propose a weighted sum of reconstruction loss together with Product of Experts like posterior model.\n  ", "summary_of_the_review": "The paper is an informative piece of work on multimodal VAE. Although, it reads like a survey paper with added suggestions. My main concern is the scalability of the approach which mixture models are meant to be efficient on.", "strength_and_weaknesses": "\nThe paper is very well written and easy to follow. I have the following comments and suggestions.\n\nComments:\n- How is the unimodal posterior $q_\\theta(z|X)$ computed and sampled from? The main motivation of mixture models is to be scalable, i.e., they can be used for large number of modalities just as effectively. The equations described in (5) and (7) look like something that is not going to scale very well with the number of modalities. Is it possible to show how the approach scales as the modality increase, empirically? \n- The paper reads more like a survey. It would have been much better if the Sections from Section 2. up to Subsection 3.3 are condensed and the contributions, Section 3.3 and Section 3.4, are expanded upon.\n- Related works is not specialized to this work-- there is a paragraph about transformer based multimodal models.\n\nSuggestions:\n- Please include what $H(\\cdot)$ actually is, this is below Eq(4). It is much better to define it there than forcing the reader to scroll through the reference. \n- Please improve result reporting in Table-1. The abbreviated dataset description and their categorization is not clear.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "phCj4QiyG5y", "reviewer": "Reviewer_Jrda", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper suggests modifying the variational objective in mixture-based multi-modal VAEs by including both a uni-modal as well as a multi-modal reconstruction term. The authors also aim to present a unified perspective on mixture-based multimodal VAEs by showing that methods motived by a Jensen-Shannon-Divergence or Total Correlation objective can be viewed as mixture-based VAEs under some assumptions. Empirical results indicate the proposed approach improves the tradeoff of generative quality versus cross-coherence for different multi-modal VAEs.\n\n\n---\nPost-rebuttal comments:\nHaving read the other reviews and the authors' response, I feel that the submission is stil borderline.\nThe authors response clarified the comments that I had previously.\nThe presentation in the updated version has been improved.\n\n---", "summary_of_the_review": "The suggested approach of including both uni- and multi-modal reconstruction terms is novel as far as I am aware. It appears to be a rather small adjustment algorithmically compared to previous work yet it can yield improved performance, particularly with respect to the cross-modal coherency.", "strength_and_weaknesses": "Strengths:\n- The idea to include the uni-modal reconstruction terms appears well motivated.\n- The empirical performance suggests that the new approach yields better generative quality (e.g., FID scores) or cross-coherence (e. g., classification accuracy) than most previous approaches.\n- The paper adds some interesting connection between MVTCAE and JMVAE and mixture-based multi-modal VAEs.\n\nWeaknesses:\n- While the suggested approach improves the cross-coherence compared to previous work, the generation quality does not improve necessarily compared to some methods.\n- The algorithmic change to include the unimodal reconstruction terms is somewhat small. \n\nComments:\n- To clarify Figure 2, do the different number of modalities mean that a different generative model is trained for different number of modalities, or is this the same generative model that has a different number of missing modalities when encoding the latents?\n- Does Lemma 1 assume a MoE approximate posterior aggregation as in Lemma 2?\n- Does the approach give good accuracy to classify the labels for PolyMNIST/Translated-PolyMNIST in the latent space?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dTRtrszLTA", "reviewer": "Reviewer_P3oY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper deals with multimodal VAEs. In particular it tries to mitigate the current limitations of models in this class. After highlighting the shortcomings of mixture-based approaches, which have been uncovered in previous work, the authors investigate the relationship between two previously proposed product-based multimodal VAEs, namely MVTCAE and MMJSD, and categorize them as coordination-based approaches. Coordination-based multimodal VAEs assume a product-of-experts joint posterior to have high-quality generated samples, and introduce specific terms in the objective to bring unimodal and joint encoders closer together, to obtain coherence between modalities. The authors propose CRMVAE, by extending the MVTCAE to include unimodal reconstruction terms in the objective, and show empirical results in multiple experiments indicating an overall improvement over existing approaches in terms of coherence and quality of generation.", "summary_of_the_review": "The paper is well-written, deals with a relevant problem, and presents promising empirical results. The authors propose a model in the class of coordination-based multimodal VAEs that extends MVTCAE, by adding unimodal reconstruction terms in the objective. However, behind this addition, there only seems to be an intuitive argument. While empirical results are promising, reproducibility can be improved, and more empirical results could give a clearer picture on whether limitations of existing multimodal VAEs are fully addressed.", "strength_and_weaknesses": "**Strengths**: \n- The paper investigates a relevant problem, trying to overcome the limitations of existing multimodal VAEs.\n- The paper is well-written and well-structured.\n- The Experiments section shows the authors have made an effort to evaluate their proposed method on a wide range of datasets, and empirical results indicate a clear improvement over previous methods.\n\n**Weaknesses**:\n\n- The main contribution of the authors when proposing the CRMVAE is introducing unimodal reconstruction terms in the objective. However, the rationale behind this choice seems to be rather based on intuition and lacks theoretical grounding, leaving some open questions. It is e.g. unclear why optimizing for unimodal reconstruction of each modality, i.e. reconstruction of each modality given the latent code inferred from the corresponding unimodal encoder, would improve coherence for cross-generation.\n- The experimental results are extensive in the sense of models being evaluated on a wide range of datasets, but there would be additional results that would give important insights on whether certain current limitations of multimodal VAEs are truly overcome. For example, how do the models compare on generation performance, when the joint latent code is sampled from a prior? This is an important aspect when trying to learn a joint distribution of all modalities [1-2], and I think coherence across generated modalities in this setting would show if bringing unimodal and joint posteriors closer together effectively leads to encoding shared content effectively in the latent space.\n- From my understanding it is unclear if the equality in section 3.4 below equation 7, central to deriving the ELBO, is justified for the considered setting. Could the authors elaborate on that? Also, it seems like the assumption could be stated more clearly: Are the integrals actually necessary for the left-hand side, or is it equivalent to assuming that the joint posterior is equal to each unimodal posterior? Can the assumption be stated in terms of a (conditional) independence relation? \n- Another important point where this work can in my opinion be improved is reproducibility. Being quite familiar with the mentioned models and experimental settings, I would have liked to replicate some of the results shown in the paper, but the authors did not share their code, which makes it hard to do so. In addition, I tried to reproduce the results for MVTCAE in the CUB Image-Captions experiment, using the public repo for the original paper, but I was not able to obtain the same performance shown in the paper for cross-generation, despite using the same settings. I think it would be helpful to see the code to understand the implementation better and reproduce the results. \n- [Minor] I believe a logarithm is missing in the RHS of the equation in Proof 1. \n\n**References**\n- [1] Yuge Shi, et al.: Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models. NeurIPS 2019.\n- [2] Thomas M. Sutter, et al.: Multimodal Generative Learning Utilizing Jensen-Shannon-Divergence. NeurIPS 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XBZ7TPoYF0", "reviewer": "Reviewer_DC9h", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "It is challenging to infer a joint representation from arbitrary subsets of multimodalities, and the state-of-the-art approaches (mixture-based multimodal VAEs) attempt to accomplish this by training to generate all modalities from a joint representation inferred from missing modalities, but the quality of modality generation is lower than that of unimodal VAEs, and this limitation is theoretically unavoidable.  Therefore, the authors propose a coordination-based model that brings the representation inferred from each modality closer to that inferred from all modalities. Experiments with diverse and challenging datasets show the advances of the proposed method.", "summary_of_the_review": "Unfortunately, this paper lies outside of my field of expertise, therefore, my comment may be biased. Area chairs are suggested to seek opinions from other reviewers.", "strength_and_weaknesses": "\nStrength:\n\n- The authors point out why these existing coordinate-based models perform poorly on cross-model generation, and propose a novel model to fix the issue.\n- The proposed model can mitigate the limitations in multimodal VAEs and performs well in both cross-coherence and generation quality.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject