On the Limitations of Multimodal VAEs
Authors: Imant Daunhawer
Thomas M. Sutter
Kieran Chin-Cheong
Emanuele Palumbo
Julia E Vogt
Venue: ICLR 2022 Poster (2022)
Keywords: multimodal learning
variational autoencoder
variational information bottleneck
information theory

Abstract: Multimodal variational autoencoders (VAEs) have shown promise as efficient generative models for weakly-supervised data. Yet, despite their advantage of weak supervision, they exhibit a gap in generative quality compared to unimodal VAEs, which are completely unsupervised. In an attempt to explain this gap, we uncover a fundamental limitation that applies to a large family of mixture-based multimodal VAEs. We prove that the sub-sampling of modalities enforces an undesirable upper bound on the multimodal ELBO and thereby limits the generative quality of the respective models.  Empirically, we showcase the generative quality gap on both synthetic and real data and present the tradeoffs between different variants of multimodal VAEs. We find that none of the existing approaches fulfills all desired criteria of an effective multimodal generative model when applied on more complex datasets than those used in previous benchmarks. In summary, we identify, formalize, and validate fundamental limitations of VAE-based approaches for modeling weakly-supervised data and discuss implications for real-world applications.

Published as a conference paper at ICLR 2022
ON THE LIMITATIONS OF MULTIMODAL VAES
Imant Daunhawer, Thomas M. Sutter, Kieran Chin-Cheong, Emanuele Palumbo & Julia E. Vogt Department of Computer Science ETH Zurich dimant (aant.eehz.cc
ABSTRACT
Multimodal variational autoencoders (VAEs) have shown promise as efficient gen- erative models for seakly-superrised data. Yet, despite their advantage of weak supervision, they exhibit a gap in generative quality compared to unimodal VAEs, which are completely unsupervised. In attempt to explain this gap, we un- cover a fundamental limitation that applies to a large family of mixture-basee multimodal VAEs. We prove that the sub-sampling of modalities enforces unde- sirable upper bound on the multimodal ELBO and thereby limits the generative quality of the respective models. Empirically, gap on both synthetic and real data and present the tradeoff's between different vari- ants of multimodal VAEs. We find that none of the existing approaches fulfills all desired criteria of an effective multimodal generative model when applied on more complex datasets than those used previous benchmarks. In summary. we identify, formalize, and validate fundamental limitations of VAE-based approaches for mod- eling weakly-supervised data and discuss implications for real-world applications.
showcase the generative quality
INTRODUCTION
In recent years, multimodal VAEs have shown great potential as efficient generative models for weakly-supervised data, such as pairs of images or paired images and captions. Previous works (Wu and Goodman, 2018; Shi et al., 2019; Sutter et al., 2020) demonstrate that multimodal VAEs leverage weak supervision to learn generalizable representations, useful for downstream tasks (Dorent et al., 2019; Minoura et al., 2021) and for the conditional generation of missing modalities (Lee and van der Schaar, 2021). However, despite the advantage of weak supervision, -tate-of-theeart multimodal VAES consistently underperform when compared to simple unimodal VAEs in terms of generative quality. This paradox serves a starting point for our work, which aims explain the observed lack of generative quality in terms of a fundamental limitation that underlics existing multimodal VAEs.
What is limiting the generative quality of multimodal VAEs? We find that the sub-sampling of modalities during training leads to problem that affects all mixture- based multimodal VAEs- a family of models that subsumes the MMVAE (Shi et al., 2019), MoPoE- VAE (Sutter al., 2021), and a special case of the MVAE (Wu and Goodman, 2018). We prove that modality sub-sampling enforces an undesirable upper bound on the multimodal ELBO and thus prevents tight approximation of the variation in the data. Our experiments demonstrate
joint distribution when there is modality-specine that modality sub-sampling can explain the gap in generative quality compared unimodal VAEs and that the gap typically increases with each additional modality. Through extensive ablations on three different datasets, we validate the generative quality gap between unimodal and multimodal VAES and present the tradeoffs between different approaches.
Our results raise serious concerns about the utility of multimodal VAEs for real-world applications. We show that none of the existing approaches fulfills all desired criteria (Shi et al., 2019; Sutter et al., 2020) of an effective multimodal generative model when applied to slightly more complex datasets than used in previous benchmarks. In particular, we demonstrate that generative coherence (Shi et al., 2019) cannot be guaranteed for any of the existing approaches, if the information shared between modalities cannot be predicted in expectation across modalities. Our findings particularly relevant for applications on datasets with a relatively high degree of modality-specifet variation, which is a typical characteristic of many real-world datasets (Baltrušaitis al., 2019).
'The lack of generative quality can even he recognized by visual inspection of the qualitative results from previous works; for instance, the supplementaries of Sutter al. (2021) or Shi al. (2021).

---

Published as a conference paper at ICLR 2022
2 RELATED WORK
First, to put multimodal VAEs into context, let us point out that there is a long line of research focused on learning multimodal generative models based on a wide variety of methods. There are several notable generative models with applications on pairs of modalities (e.g., Ngiam al., 20 Srivastava and Salakhutdinov, 2014; Wu and Goodman, 2019; Lin et al.. 2021; Ramesh et al., 2021), as well as for the specialized task of image-to-image translation (c.g. Huang et al., 2018; Choi et al., 2018; Liu et al., 2019). Moreover, generative models can use labels as side information (Ilse et al., 2019; Tsai et al., 2019; Wieser et al., 2020); for example, to guide the disentanglement of shared and mopality-peecific information (Tsai et al., 2019). In contrast, multimodal VAEs do not require strong supervision and can handle a large and variable number modalities efficiently. They learn joint distribution over multiple modalities, but also enable the inference of latent representations, as well as the conditional generation of missing modalities, given any subset of modalities (Wu and Goodman, 2018; Shi et al., 2019; Sutter et al., 2021).
Multimodal VAEs are an extension of VAEs (Kingma and Welling, 2014) and they belong to the class of multimodal generative models with incodr-ddccoedd architectures (Baltrušaitis et al., 2019). The first multimodal extensions of VAES (Suzuki et al., 2016; Hsu and Glass, 2018; Vedantam et al., 2018) use separate inference networks for every subset of modalities, which quickly becomes intractable as the number of inference networks required grows exponentially with the number of modalities. Starting with the seminal work of Wu and Goodman (2018), multimodal VAEs were developed as an efficient method for multimodal learning. In particular, multimodal VAEs enable the inference of latent representations. as well as the conditional generation of missing modalities, given any subset of input modalities. Different types of multimodal VA were devised by decomposing the joint encoder as a product (Wu and Goodman, 2018), mixture (Shi et al., 2019), or mixture of products (Sutter et al., 2021) 1) of unimodal encoders respectively. A commonality between these approaches is the sub-sampling of modalities during training- a property we will use to define the family of mixture-based multimodal VAEs. For the MMVAE and MoPoE-VAN the sub-sampling is direct consequence of defining the joint encoder as a mixture distribution over different subsets of modalities. Further, our analysis includes special case of the MVAE without ELBO sub-sampling which can be seen as another member of the family of mixture-basse based multimodal VAES (Sutter et al., 2021). I). The MVAE was originally proposed with "ELBO sub-sampling" that was later found to result in an incorrect bound on the joint distribution (Wu and Goodman, 2019). While this training paradigm is also based on the sub-sampling of modalities, the objective differs from mixture-based multimodal VAEs in that the MVAE does not reconstruct the missing modalities from the set of sub-sampled modalities.?
additional training paradigm
Table provides an overview of the different variants of mixture- multimodal VAEs and the properties that one can infer from empirical results in previous works (Shi et al., 2019; 2021: Sutter et al., 2021). Most importantly, there appears to be a tradeoff between generative quality and generative coherence (i.c., the ability to generate semantically related samples across modalities). Our work explains why the generative quality is worse for models that sub-sample modalities (Section 4) and shows that a tighter approximation of the joint distribution can be achieved without sub-sampling (Section 4.3). Through systematic ablations, we validate the proposed theoretical limitations and showcase the tradeoff between generative quality and generative coherence (Section 5.1). Our experiments also reveal that generative coherence cannot be guaranteed for more complex datasets than those used in previous benchmarks (Section 5.2).
3 MULTIMODAL VAES, IN DIFFERENT FLAVORS
Let X = (Xi,...,Xar) be set of random vectors describing M modalities and let x = [Z),.... DM) be a sample from the joint distribution p(z1,.... .MM For conciseness, denote subsets of modalities by subscripts; for example, X (1,33 or a respectively for modalities 1 and 3.
Throughout this work, we assume that all modalities are described by discrete random vectors (c.g.. pixel values), that we can assume non- negativee entropy and conditional entropy terms. Definitions for all required information-theonm theoretic quantities are provided in Appendix A.
For completeness, in Appendix C, we also analyze the effect of ELBO sub-sampling.
2

---

Published as a conference paper at ICLR 2022
Table 1: Overview of multimodal VAEs. Entries for generative quality and generative coherence denote properties that were observed empirically in previous works. The lightning symbol (á denotes properties for which our work presents contrary evidence. This overview abstracts technical details, such importance sampling and ELBO sub-sampling, which we address in Appendix C.
Model
Decompostion of (a a)
Modality sub sampling
Generatiee quality
Generative cohfrennce
MVAE (Wa and Goodman, 2018)
I Pu( [())
good
MMVAE (Shi al., 2019)
22 EM pola mmg)
/
Timited
good
MoPoF-VAF (Soteer al., 2021)
P POr EaeP(ar) The ro(x)m))
limited
good
3.1 THE MULTIMODAL ELBO
Definition 1. Let pa (z)( x)) be a stochastic encoder, parameterized by 0, that takes multiple modalities as input. Let qo(2 (æ 2) be variational decoder (for all modalities), parameterized by d, and let q(z) be a prior. The multimodal evidence lower bound (ELBO) on Eppze p(æ:) is defined as
((x; 0, 0) = Ep(arpo(== un) [log 96 x)] Ep(=) [Kx (pa x) [[)())))]
(1)
The multimodal ELBO (Definition 1), first introduced by Wu and Goodman (2018), is the objective maximized by all multimodal VAEs and it forms variational lower bound on the expected log- evidence. 3 The first term denotes the estimated log-likelihood of all modalities and the second term the KL-divergence between the stochastic encoder and the prior. We take an information-theoree perspective using the variational information bottleneck (VIB) from Alemi et al. (2017) and employ the standard notation used in multiple previous works (Alemi et al., 2017; Poole et al., 2019). Similar to the latent variable model approach, the VIB derives the ELBO as a variational lower bound on the expected log-evidence, but, in addition, the VIB is a more general framework for optimization that allows us reason about the underlying information-theorede quantities of interest (for details on the VIB and its notation, please see Appendix B.
Note that the above definition of the multimodal ELBO requires that the complete set of modalities is available. To overcome this limitation and to learn the inference networks for different subsets of modalities, existing models use different decompositions of the joint encoder. as summarized in Table 1. Recent work shows that existing models can be generalized by formulating the joint encoder as mixture of products of experts (Sutter et al., 2021). Analogously, in the following, we generalize existing models to define the family of mixture- based multimodal VAEs.
3.2 THE FAMILY OF MIXTURE-BASED MULTIMODAL VAES
Now we introduce the family of mixture-based multimodal VAEs, which subsumes the MMVAE, MoPoE-VAE, and special case of the MVAE without ELBO sub-sampling, We first define an encoder that generalizes the decompositions used by existing models:
Definition 2. Let S = ((A,w>) A S (I...., M), A + 0, WA E 1]} be an arbitrary of non- empty subsets A of modalities and corresponding mixture coefficients W.S such that E WA = 1.
Define the stochastic encoder to be a mixture distribution: pp(z x) = -AAS WA Po(z A))-
In the above definition and throughout this work, we write A E S to abbreviate (A, E S. To define the family of mixture- based multimodal VAES, we restrict the family of models optimizing the multimodal ELBO to the subfamily of models that use a mixture-basee stochastic encoder.
Definition 3. The family of mixture- based multimodal VAEs is comprised of all models that maximize the multimodal ELBO using a stochastic encoder Pe (z x) that is consistent with Definition 2. In particular, we define the family terms of all models that maximize the following objective:
Cs(x; Σav WA (ea(1Ep (Ep(s)po(s a Ilog qo =)] Ep(() (DRL (po(z (x 1l {[)))]]] (2)
AES
'Even though we write the expectation over p(æ). for the estimation of the ELBO we still assume that we only have access to finite sample from the training distribution p(x). The notation is used for consistency with the well-established information-thcome perspective on VAEs (Alemi al., 2017; Poole al., 2019).
3

---

Published as a conference paper at ICLR 2022
In Appendix B.2, we show that the objective Ls(=; 8 o) is lower bound on L(x; 0, Φ) (which makes it an ELBO) and explain how, for different choices of the set subsets S, the objective Ls (z; f, o) relates to the objectives of the MMVAE, NoPoE-VAE. and MVAE without ELBO sommmiillig....
From a computational perspective, a characteristic of mixture-based multimodal VAEs is the sub- sampling of modalities during training, which is direct consequence of defining the encoder mixture distribution over subsets of modalities. The sub-sampling of modalities can be viewed as the extraction of a subset a A E æt where A indexes one subset of modalities that is drawn from the model-specific set of subsets S. The only member of the family of mixture-based multimodal VAEs that forgoes sub-sampling, defines a trivial mixture over single subset, the complete set of modalities (Sutter et al., 2021).
4 MODALITY SUB-SAMPLING LIMITS THE MULTIMODAL ELBO
4.1 AN INTUITION ABOUT THE PROBLEM
Before we delve into the details, let us illustrate how modality sub-sampling affects the likelihood esti- mation, and hence the multimodal ELBO. Consider the likelihood estimation using the objective Ls:
Σ WA Ep(ar)os( aa) log (x [z]] AES
(3)
where A denotes a subset of modalities and ωA the respective mixture weight. Crucially, the stochastic encoder po aA encodes subset of modalities. What seems to a minute detail, can have profound impact on the likelihood estimation, because the precise estimation of all modalities depends on information from all modalities. In trying to reconstruct all modalities from incomplete information, the model can learn an inexact, average prediction; however, it cannot reliably predict modality speciffc information, such as the background details in an image given a concise verbal description of its content.
In the following, we formalize the above intuition by showing that, in the presence of modality- specific variation, modality sub-sampling enforces undesirable upper bound on the multimodal ELBO and therefore prevents a tight approximation of the joint distribution.
4.2 A FORMALIZATION OF THE PROBLEM
Theorem states our main theoretical result, which describes non non-trivial limitation of mixture- multimodal VAEs. Our result shows that the sub-sampling of modalities enforces an undesirable upper bound on the approximation of the joint distribution when there is modality-speciff variation in the data. This limitation conflicts with the goal of modeling real-world multimodal data, which typically exhibits a considerable degree modality-specife variation.
Theorem 1. Each mixture-based multimodal VAE (Definition 3) approximates the expected log- evidence up to an irreducible discrepancy ( S) that depends on the model-specific mixture distribution S well as on the amount of modality-specific information in X.
For the maximization of Lss (2; 0, Φ) and every value of 0 and 6. the following inequality holds: Ep(an) (log:(l)))) p(x)) 2 Ls(x; 0, 6) + S(X,S)
(4)
where
A(X.S) - (X..((X (I.... (XA) AES
(5)
In particular, the generative discrepancy is always greater than or equal to zero and it is independent of and & and thus remains constant during the optimization.
A proof is provided in Appendix B.5 and it is based on Lemmas 1 and 2. Theorem 1 formalizes the rationale that, in the general case, cross-modal prediction cannot recover information that is specific to the target modalities that are unobserved due modality sub-sampling. In general, the conditional entropy H(X
(A)) measures the amount of information in one subset of random vectors X (I)...MYYA A that is not shared with another subset XA. In our context, the sub-sampling of modalities yields discrepancy A A(X,S) S) that is weighted average of conditional
4

---

Published as a conference paper at ICLR 2022
entropies H(X, (1 (I....MAAA Xa) observed subset XA. Hence, A ((X; S)) describes the modality-specifie information that cannot be recovered by cross-modal prediction, averaged over all subsets of modalities.
of the modalities X (I....,MY MYA unobserved by the encoder given an
Theorem applies to the MMVAE, MoPoE- and special case of the MVAE without ELBO sub- sampling, since all of these models belong to the class of mixture-based multimodal VAEs. However, A(X,S) can vary significantly between different models, depending the mixture distribution defined by the respective model and on the amount of modality-specifff variation in the data. In the following, we show that without modality sub sampling A S) vanishes, whereas for the MMVAE and PoPoF-VAE, A ( S) typically increases with each additional modality. In Section 5, we provide empirical support for each of these theoretical statements.
4.3 IMPLICATIONS OF THEOREM 1
First, we consider the case of no modality sub-sampling, for which it is easy to show that the generative discrepancy vanishes. Corollary 1. Without modality sub-sampling, A(X, 0.
A proof is provided in Appendix B.6. The result from Corollary I applies to the MVAE without ELBO sub-sampling and suggests that this model should yield a tighter approximation of the joint distribution and hence better generative quality compared to mixture- based multimodal VAEs that sub-sample modalities. Note that this does not imply that a model without modality sub-sampling is superior to one that uses sub-sampling and that there can be an inductive bias that favors sub-sampling despite the approximation error it incurs. Especially, Corollary I does not imply that the variational approximation is tight for the MVAE; for instance, the model can be underparameterrized simply misspecified due to simplifying assumptions, such as the PoE- foo-ooactoinztton (Kurle et al., 2019).
Second, we consider how additional modalities might affect the generative discrepancy. Corollary predicts an increased generative discrepancy (and hence, a decline of generative quality) when we increase the number of modalities for the MMVAE and MoPoE- VAE.
Corollary (informal). For the MMVAE and MoPoE-VAE: the generative discrepancy increases with each additional modality, if the new modality is sufficiently diverse.
A proof is provided in Appendix B.7. The notion of diversity requires a more formal treatment of the underlying information-thheortie quantities, which we defer to Appendix B.7. Intuitively, new modality is sufficiently diverse, it does not add too much redundant information with respect to the existing modalities. In special cases when there is a lot of redundant information, A(X, S) can decrease given an additional modality. but it does not vanish in any one of these cases. Only if there very little modality-specifee information in all modalities, we have A(X, S) 0 for the MMVAE and MoPoF- VAF. This condition requires modalities to be extremely similar, which does not apply to most multimodal datasets, where A(X, S) typically represents large part of the total variation.
In summary, Theorem formalizes how the family of mixture-based multimodal VAEs is fundamen- tally limited for the task of approximating the joint distribution, and Corollaries and connect this result to existing models-ttee MMVAE, MoPoE- VAE, and MVAE without ELBO sabbsampling.. We now turn to the experiments, where we present empirical support for the limitations described by Theorem and its Corollaries.
5 EXPERIMENTS
Figure presents the three considered datasets. PolyMNIST (Sutter et al., 2021) is a sim- ple, synthetic dataset with five image modalities that allows us to conduct systematic ablations. Translated-Polyyne PolyMNIST is a new dataset that adds small tweak-the the downscaling and random translation digits-to demonstrate the limitations existing methods when shared information cannot be predicted in expectation across modalities. Finally, Caltech Birds (CUB; Wah et al., 2011; Shi et al., 2019) is used to validate the limitations on a more realistic dataset with two modalities, images and captions. Please note that use CUB with real images and not the simplified version based on precomputed ResNet-features that was used in Shi et al. (2019) and Shi et al. (2021). For a more detailed description of the three considered datasets, please see Appendix C.I.
5

---

Published as a conference paper at ICLR 2022
2
Z
4
6
7
9
2
3
4
5
4
5
9
N0% rdde
(a) PolyMNIST
(b) Translated PollMMMII (5 modalities)
(c) Caltech Birds (CUB) (2 modalities)
(5 modalities)
Figure 1: The three considered datasets. Each subplot shows samples from the respective dataset. The two PolyMNIST datasets are conceptually similar in that the digit label is shared between five synthetic modalities. The Caltech Birds (CUB) dataset provides more realistic application for which there is no annotation on what is shared between paired images and captions.
In total, more than 400 models were trained, requiring approximately 1.5 GPU years of compute on a single NVIDIA GeForce RTX 2080 Ti GPU. For the experiments in the main text, we use the publicly available code from Sutter et al. (2021) and in Appendix C.3 we also include ablations using the publicly available code from Shi et al. (2019), which implements importance sampling and alternative ELBO objectives. To provide a fair comparison across methods, we use the same architectures and similar capacities for all models. For each unimodal VAE, make sure decrease the capacity by reducing the latent dimensionality proportionally with respect to the number of modalities. Additional information on architectures, hyperparameterrs and evaluation metrics is provided in Appendix C.
5.1 THE GENERATIVE QUALITY GAP
We assume that an increase in the generative discrepancy A(X, S) is associated with a drop of generative quality. However, we want point out that there can also be an inductive bias that favors modality sub-sampling despite the approximation error that it incurs. In fact, our experiments reveal a fundamental tradeoff between generative quality and generative coherence when shared information can be predicted in expectation across modalities.
We measure generative quality in terms of Fréchet inception distance (FID; Heusel et al., 2017), standard metric for evaluating the quality of generated images. Lower FID represents better generative quality and the values typically correlate well with human perception (Borji, 2019). In addition, in Appendix C we provide log-likelihood values, as well as qualitative results for all modalities including captions, for which FID cannot be computed.
Figure 2 presents the generative quality across range of 3 values.* To relate different methods, we compare models with the best FID respectively, because different methods can reach their optima at different B3 values. As described by Theorem 1, mixture--asee multimodal VAEs that sub-sample modalities (MMVAE and MoPoE-VALE exhibit pronounced generative quality gap compared to unimodal VAEs. When compare the best models, observe gap of more than 60 points both PolyMNIST and Translated- PolyaedS,,,, and about 30 points on CUB images. Qualitative results (Figure 9 in Appendix C.3) confirm that this gap is clearly visible in the generated samples and that it applies not only to image modalities, but also to captions. In contrast, the MVAE (without ELBO sub-sampling) reaches the generative quality of unimodal VAEs, which is in line with our theoretical result from Corollary 1. For completeness, in Appendix C.3, we also report joint logelikelsooods, latent classification performance, as well as additional FIDs for all modalities.
Figure 3 examines how the generative quality is affected when we vary the number of modalities. Notably, for the MMVAE and MoPoE-VAIE the generative quality deteriorates almost continuously with the number of modalities, which is in line with our theoretical result from Corollary 2. Interest- ingly, for the MVAE, the generative quality on Translatdd-PolyMNI the change is comparatively small. Figure 11 in Appendix C.3, shows a similar trend even when we control for codalitt-sspciifi differences by generating PolyMNIST using the same background image for all modalities.
also decreases slightly, but
"The regularization coefficient 8 weights the KI. -iveeergee term of the multimodal EIBO (Definitions 1 and 3) and it is arguably the most impactful hyperparameter in VAFs (e.g., Higgins et al., 2017).

---

Published as a conference paper at ICLR 2022
392
500
-901
MMVAE McPUE VAE
MMVAE
200
350
McRLE-VAE
200
unimodal VAE
200
unimodal VAE
110
100
MMVAE
60-
McPoE-VAE
VAN
0-
(a) PolyMNIST
(b) Translated-Poly
(c) Caltech Birds (CUB)
Figure 2: Generative quality for one output modality over a range of 3 values. Points denote the FID averaged over three seeds and bands show one standard deviation respectively. Due to numerical instabilities, the MVAE could not be trained with larger B values.
200
200
175
MoPuE-VAE
MVAE MMVAE
McPoE VAE
MMVAE
unimodal
unimodal VAE
125-
125
100
50-
64-
75
85
number of modalities
number of modalities
(a) PolyMNIST
(b) Tralslated-PlyyMNISTT
Figure 3: Generative quality as a function of the number of modalities. The results show the FID of the same modality and therefore all values the same scale. All models are trained with PolyMNIST and 8 = 0.3 on TranslaeddPoolyMie the bands show one standard deviation respectively. For the unimodal VAE, which uses only single modality, the average and standard deviation are plotted as a constant.
on The results are averaged over three seeds and
In summary, the results from Figure 2 and Figure 3 provide empirical support for the existence of a generative quality gap between unimodal and mixture- based multimodal VAEs that sub-sample modalities. The results verify that the approximation of the joint distribution improves for models without sub-sampling, which manifests in better generative quality. In contrast, the gap increases disproportionally with each additional modality for both the MMVAE and MoPoE-VAE, Hence, the presented results support all of the theoretical statements from Sections 4.2 and 4.3.
5.2 LACK OF GENERATIVE COHERENCE ON MORE COMPLEX DATA
Apart from generative quality, another desired criterion (Shi et al., 2019; Sutter et al., 2020) for an effective multimodal generative model is generative coherence, which measures model's ability to generate semantically related samples across modalities. To be consistent with Sutter et al. (2021), we compute the leave-one-out coherence (see Appendix C.2). which means that the input to each model consists of all modalities except the one that is being conditionally generated. On CUB, we resort to a qualitative evaluation of coherence, because there is no ground truth annotation of shared factors and the proxies used in Shi et al. (2019) and Shi et al. (2021) do not yield meaningful estimates when applied to the conditionally generated images from models that were trained on real images.
In terms of generative coherence. Figure 4 reveals that the positive results from previous work do not translate to more complex datasets. As a baseline, for PolyMNIST (Figure 4a) we replicate the coherence results from Sutter et al. (2021) for a range of 3 values. Consistent with previous work (Shi et al., 2019; 2021; Sutter et al., 2020; 2021), we find that the MMVAE and MoPoE- VAE exhibit
Pleasee note that previous work (Shi et al., 2019; 2021) used simplified version of the CUB dataset, where images were replaced by precomputed Restuctfcatures.
7

---

Published as a conference paper at ICLR 2022
Conditional Coherence Accuracy
9.8-
0.5-
MVAE MMWAE
0.4
MoPoE-VAE
0.2-
MVAE, 3 9
0.0
10a
(a) PolyMNIST
Conditional Coherence Accuracy
MVAE
0.8-
MMWWE
MMVAE, 3 =9
MaPoE-VA VAE
3.6-
0.4-
0.0
10
10
10
10P
10
MoPoE-VAE, ==
(b) Translated-Polye PolyMNIST
(c) Caltech Birds (CUB)
Figure 4: Generative coherence for the conditional generation across modalities. For PolyMNIST (Figures 4a and 4b), we plot the average leave-one-out coherence. Due numerical instabilities, the MVAE could not be trained with larger 3 values. For CUB (Figure 4c), we show qualitative results for the conditional generation of images given captions. Best viewed zoomed and color.
superior coherence compared to the MVAE. Though, it was not apparent from previous work that MVAE's coherence can improve significantly with increasing / values, which can be of independent interest for future work. On Monislated-PolMMNSSS (Figure 4b), the stark decline of all models makes it evident that coherence cannot be guaranteed when shared information cannot be predicted expectation across modalities. Our qualitative results (Figure 10 Appendix C.3) confirm that not single multimodal VAE is able to conditionally generate coherent examples and, for the most part, not any digits at all. To verify that the lack of coherence is not an artifact of our implementation, we have checked that the encoders and decoders have sufficient capacity such that digits show up in most seff-reconstructions. On CUB (Figure 4c), for which coherence cannot be computed, the qualitative results for conditional generation verify that none of the existing approaches generates images that both of sufficiently high quality and coherent with respect to the given caption. Overall, the negative and CUB showcase the limitations of existing approaches when applied to more complex datasets than those used in previous benchmarks.
results on Translated-PolyMMe
6 DISCUSSION
Implications and scope Our experiments lend empirical support to the proposed theoretical limi- tations of mixture-based multimodal VAEs. On both synthetic and real data, our results showcase the generative limitations of multimodal VAEs that sub-sample modalities. However, our results also reveal that none of the existing approaches (including those without sub-sampling) fulfill all desired criteria (Shi et al., 2019; Sutter et al., 2020) of an effective multimodal generative model. More broadly, our results showcase the limitations of existing VAE-based approaches for modeling data in the presence of modality-speciff information, and in particular when shared information cannot be predicted in expectation across modalities. The Translated Translated-PolyMNNIS PolSMNIST dataset demonstrates this problem in simple setting, while the results CUB confirm that similar issues can expected on more realistic datasets. For future work, it would be interesting to generate simulated data where the discrepancy A A(X, S) be measured exactly and where it is gradually increased by an adaptation the dataset in a way that increases only the moodality-specific variation. Furthermore, it is worth noting that Theorem applies to all multimodal VAFs that optimize Equa-
8

---

Published as a conference paper at ICLR 2022
tion (2), which is lower bound on the multimodal ELBO for models that sub-sample modalities. Our theory predicts the same discrepancy for models that optimize a tighter bound (e.g., via Equation (28)), because the discrepancy A(X, , S) derives from the likelihood term, which is equal for Equations (2) and (28). In Appendix C.3 we verify that the discrepancy can also be observed for the MMVAE with the original implementation from Shi al. (2019) that uses tighter bound. Further analysis of the different bounds can be an interesting direction for future work.
Model selection and generalization Our results raise fundamental questions regarding model selection and generalization, as generative quality and generative coherence do not necessarily go hand in hand. In particular, our experiments demonstrate that FIDs and log- ligelikilioods do not reflect the problem of lacking coherence and without access to ground truth labels (on what is shared between modalities) coherence metrics cannot be computed. As a consequence, it can be difficult to perform model selection more realistic multimodal datasets, especially for less interpretable types of modalities, such as DNA sequences. Hence, for future work it would be interesting to design alternative metrics for generative coherence that can be applied when shared information is not annotated. For the related topic of generalization, it can be illuminating to consider what would happen, if one could arbitrarily "scale things up". In the limit of infinite i.i.d. data, perfect generative coherence could be achieved by model that memorizes the pairwise relations between training examples from different modalities. However, would this yield a model that generalizes out of distribution (e.g., under distribution shift)? We believe that for future work it would be worthwhile to consider out-of-distribution generalization performance (e.g., Montero et al., 2021) addition to generative quality and coherence.
Limitations In general, the limitations and tradeoffss presented in this work apply to a large family of multimodal VABs, but not necessarily to other types of generative models, such generative adversarial networks (Goodfellow et al., 2014). Where current VAES are limited by the reconstruc- tion of mopality-ppecifie information, other types of generative models might offer less restrictive objectives. Similar to previous work, we have only considered models with simple priors, such as Gauss and Laplace distributions with independent dimensions. Further, we have not considered models with modality-speciff latent spaces, which seem to yield better empirical results (Hsu and Glass, 2018; Sutter al., 2020; Daunhawer et al., 2020), but currently lack theoretical grounding. Modality latent spaces offer potential solution to the problem of cross-modal prediction by providing modality-specine context from the target modalities to each decoder. However, more work is required to establish guarantees for the identifiability and disentanglement of shared and modality-specific factors, which might only be possible for VAEs under relatively strong assumptions (Locatello al., 2019; 2020; Gresele al., 2019; von Kügelgen et al., 2021).
7 CONCLUSION
In this work, we have identified, formalized, and demonstrated several limitations of multimodal VAEs. Across different datasets, this work revealed a significant gap in generative quality between unimodal and mixture-based multimodal VAEs. We showed that this apparent paradox can be explained by the sub-sampling of modalities, which enforces an undesirable upper bound on the multimodal ELBO and therefore limits the generative quality of the respective models. While the sub- sub-sampling of modalities allows these models to learn the inference networks for different subsets of modalities efficiently, there is a notable tradeoff in terms of generative quality. Finally, we studied two failure cases Prannlated-PolyMMNIII PolyMNIST and CUB cUt demonstrate the limitations of multimodal VAEs when applied to more complex datasets than those used in previous benchmarks.
For future work, we believe that it is crucial to be aware of the limitations of existing methods as a first step towards developing new methods that achieve more than incremental improvements for multimodal learning. We conjecture that there are at least two potential strategies to circumvent the theoretical limitations of multimodal VAEs. First, the sub-sampling of modalities can be combined with modality-peecific context from the target modalities. Second, cross-modal reconstruction terms can be replaced with less restrictive objectives that do not require an exact prediction of modality- specific information. Finally, we urge future research to design more challenging benchmarks and to compare multimodal generative models in terms of both generative quality and coherence across a range of hyperparameter values, to present the tradeoff between these metrics more transparently.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
ID and KC were supported by the SNSF grant #200021_18844 Special thanks to Alexander Marx, Nicolò Ruggeri, Maxim Samarin, Yuge Shi, and Mario Wieser for helpful discussions and/or feedback on the manuscript.
REPRODUCIBILITY STATEMENT
For all theoretical statements, we provide detailed derivations and state the necessary assumptions. For main theoretical results, present empirical support both synthetic and real data. To ensure empirical reproducibility, the results of each experiment and every ablation were averaged over multiple seeds and reported with standard deviations. All of the used datasets are either public or can be generated from publicly available resources using the code that we provide in the supplementary material. Information about implementation details, hyperparameter settings, and evaluation metrics are included in Appendix C.
REFERENCES
Alemi, A. A., Fischer, I., Dillon, V., and Murphy, K. (2017). Deep variational information bottleneck. In International Conference Learning Representations.
Baltrušaitis, T., Ahuja, C., and Morency, L.-P. (2019). Multimodal machine learning: survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2):223-443
Borji, A. (2019). Pros and cons of GAN evaluation measures. Computer Vision and Image Under- standing, 179:41-65.
Choi, Y., Choi, M., Kim, M., Ha, J.-W., Kim, S., and Choo, J. (2018). StarGAN: unified generative adversarial networks for multi-domain image-to-image translation. In Conference on Computer Vision and Pattern Recognition.
Cover, T. M. and Thomas, J. A. (2012). Elements of information theory. John Wiley & Sons.
Daunhawer, I., Sutter, T. M., Marcinkevies, R., and Vogt, J. E. (2020). Self-supervised disentangle- ment of modality-specte and shared factors improves multimodal generative models. In German Conference Pattern Recognition.
Dorent, R., Joutard, S., Modat, M., Ourselin, S., and Vercauteren, T. (2019). Hetero-modal variational encoder-decoder for joint modality completion and segmentation. Medical Image Computing and Computer Assisted Intervention, pages 74-82. Springer.
Goodfellow, I., Pouget-Abadic, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville. A., and Bengio, Y. (2014). Generative adversarial nets. In Advances in neural information processing systems.
Gresele, L., Rubenstein, P. K., Mehrjou, A., Locatello, F., and Schölkopf, B. (2019). The incomplete Rosetta stone problem: identifiability results for multi-view nonlinear ICA. In Conference on Uncertainty in Artificial Intelligence.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. (2017). GANs trained by two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems.
Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2017). beta- VAE: learning basic visual concepts with constrained variational framework, In International Conference Learning Representations.
Hsu, W.NN. and Glass, J. (2018). Disentangling by partitioning: a representation learning framework for multimodal sensory data. arXiv preprint IrXiv:1805. /264.
Huang, X., Liu, M., Belongie, S. J., and Kautz, J. (2018). Multimodal unsupervised image-to-image translation. In European Conference Computer Vision.
10

---

Published as a conference paper at ICLR 2022
Ilse, M., Tomezak, J. M., Louizos, C., and Welling, M. (2019). DIVA: domain invariant variational autoencoders. arXiv preprint arXiv: 1905:
Kingma, D. P. and Ba, J. (2015). Adam: a method for stochastic gradient descent. International Conference on Learning Representations.
Kingma, D. P. and Welling, M. (2014). Auto-encoding variational Bayes. International Conference Learning Representations.
Kurle, R., Guennemann, S., and van der Smagt, P. (2019). Multi-source neural variational inference. In AAAI Conference on Artificial Intelligence.
LeCun, Y., Bottou, L.., Bengio, Y., and Haffner, P. (1998). Gradient- learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2222
Lee, C. and van der Schaar, M. (2021). A variational information bottleneck approach to multi-omics data integration. In International Conference on Artificial Intelliencce and Statistics.
Lin, J., Men, R., Yang, A., Zhou, C., Ding, M., Zhang, Y., Wang, P., Wang, A., Jiang, L., Jia, X., et al. (2021). M6: chinese multimodal pretrainer. arXiv preprint arXiv:2103.00823
Liu, M., Huang, X., Mallya, A., Karras, T., Aila, T., Lehtinen, J., and Kautz, J. (2019). Few-shot unsupervised image-to-image translation. In International Conference Computer Vision.
Locatello, F., Bauer, S., Lucic, M., Rätsch, G., Gelly, S., Schölkopf, B., and Bachem, O. (2019). Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference on Machine Learning.
Locatello, F., Poole, B., Rätsch, G., Schölkopf, B., Bachem, O., and Tschannen, M. (2020). Weakly- supervised disentanglement without compromises. In International Conference on Machine Learning.
Minoura, K., Abe, K., Nam, H., Nishikawa, H., and Shimamura, T. (2021). A mixture-of-expee deep generative model for integrated analysis of single-cell multiomics data. Cell Reports Methods.
Montero, M. L., Ludwig, C. J., Costa, R. P., Malhotra, G., and Bowers, J. (2021). The role of disentanglement in generalisation. In International Conference Learning Representations.
Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., and Ng, A. Y. (2011). Multimodal deep learning. In International Conference on Machine Learning.
Poole, B., Ozair, S., Van Den Oord, A., Alemi, A., and Tucker, G. (2019). On variational bounds of mutual information. In International Conference on Machine Learning.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. (2021).
Zero-shot text-to-image generation. In International Conference on Machine Learning.
Shi, Y., Paige, B., Torr, P., and Siddharth, N. (2021), Relating by contrasting: a data-efficient frame- work for multimodal generative models. In International Conference on Learning Representations.
Shi, Y., Siddharth, N., Paige, B., and Torr, (2019). Variational mixture-of-experts autoencoders for multi-modal deep generative models. In Advances in Neural Information Processing Systems.
Srivastava, N. and Salakhutdinov, (2014). Multimodal learning with deep Boltzmann machines. Journal of Machine Learning Research, 15(1):2949-290 29800
Sutter, T. M., Daunhawer, I., and Vogt. J. E. (2020). Multimodal generative learning utilizing Jensen- haanen--Saamoo-diveeggeeee In Advances in Neural Information Processing Systems.
Sutter, T. M., Daunhawer, I., and Vogt. J. E. (2021). Generalized multimodal ELBO. In International Conference on Learning Representations.
Suzuki, M., Nakayama, K., and Matsuo, Y. (2016). Joint multimodal learning with deep generative models. arXiv preprint 1rXiv:/611.00800
11

---

Published as a conference paper at ICLR 2022
Tsai, Y.-H. H., Liang, P. P., Zadeh, A., Morency, L.-P., and Salakhutdinov, R. (2019). Learning factorized multimodal representations. In International Conference on Learning Representations.
Tucker, G., Lawson, D., Gu, S., and Maddison, C. J. (2019). Doubly reparameterized gradient estimators for Monte Carlo objectives. International Conference Learning Representations.
Vedantam, R., Fischer, I., Huang, J., and Murphy, K. (2018). Generative models of visually grounded imagination. In International Conference on Learning Representations.
von Kügelgen, J., Sharma, Y., Gresele, L., Brendel, W., Schölkopf. B., Besserve, M., and Locatello, F. (2021). Self-supervised learning with data augmentations provably isolates content from style. In Advances in Neural Information Processing Systems.
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. (2011). The Caltech- Birds- 200-2011 dataset. Technical Report CNS-TR 20-2TR--01111 1-001, California Institute of Technology.
Wieser, M., Parbhoo, S., Wieczorek, A., and Roth, V. (2020). Inverse learning of symmetry transfor- mations. In Advances Neural Information Processing Systems.
Wu, M. and Goodman, N. (2018). Multimodal generative models for scalable weakly-supervised learning. In Advances in Neural Information Processing Systems.
Wu, M. and Goodman, N. D. (2019). Multimodal generative models for compositional representation learning, arXiv preprint 191200055..
12

---

Published as a conference paper at ICLR 2022
A DEFINITIONS
Let A. V, and 2 denote the support sets of three discrete random vectors X, Y, and Z respectively. Let px (2)), py (y), and pz(z) denote the respective marginal distributions and note that will leave out the subscripts (e.g., p(x) instead of (x) when it is clear from context which distribution we are referring to. Analogously, we write shorthand p(y x) for the conditional distribution of Y given X and p(z: y) for the joint distribution of X and Y.
The entropy of X defined as
H(X)=
Ep(a) logp(a) æEA
(6)
The conditional entropy of X given Y is defined as
H(X Y) =- Σ
p(a, y) log y)
(7)
WE
The joint entropy of X and Y is defined as
H(X, Y) =
Σ
plæ, y) log y)
(8)
NE
The Kullback-Leibler divergence of the discrete probability distribution P from the discrete probabil- ity distribution Q is defined as
P(æ) ((x)
P(x) D (P (1 P(x) log
(9)
assuming that and Q are defined on the same support set X.
The cross-entropy of the discrete probability distribution from the discrete probability distribution P is defined as
CE(P.)) = Σ P(z) log Q(x)
(10)
REX
assuming that P and Q are defined the same support set X.
The mutual information of X and is defined as
I(x; Y) (pl(p( y) p())p(y))
(11)
The conditional mutual information of X and Y given Z is defined as
I(X;Y Y Z) = Σ (px(, |) p( xx)p(y| =))
(12)
ZEZ
Recall that we assume discrete random vectors (e.g., pixel values) and therefore can assume non- negative entropy, conditional entropy and conditional mutual information terms (Cover and Thomas, 2012). For continuous random variables, all of the above sums can be replaced with integrals. The only coommatioo-thhoreetee quantities for which this work we use continuous random vectors are the KL-divergence and mutual information, both of which are always negaaive....
13

---

Published as a conference paper at ICLR 2022
B PROOFS
B.1 INFORMATION-THEORFII DERIVATION OF THE MULTIMODAL. ELBO
Proposition I relates the multimodal ELBO (Definition 1) to the expected log-evidence, the quantity that is being approximated by all likelihood- -based generative models including VAEs. The derivation based on straightforward extension of the variational information bottleneck (VIB; Alemi et al., 2017). We include the result mainly for the purpose of illustration-ue clarify the notation, as well as the relation between the multimodal ELBO and the underlying information-theoreti quantities of interest: the entropy, conditional entropy, and mutual information.
Notation Readers who familiar with latent variable models, but may be less familiar with the information-theoreiie perspective on VAEs, please keep in mind the following notational differences. In contrast the latent variable model perspective, which defines variational posterior (typically denoted by the letter 9) and a stochastic decoder (lypically denoted by the letter p), the VIB defines stochastic encoder Po(= a)) and variational decoder (s z). Moreover, the VIB makes no assumptions about the true posterior. Also note that latent variable models tend to write the ELBO with respect to the log-evidence log p(z)). but information -hheoretti approaches write the ELBO with respect to the expected log-evidence pp) [lo p(=:)]; though, it is still assumed that the estimation of the ELBO is based on a finite sample from p(x)).
Proposition 1. The multimodal ELBO forms variational lower bound on the expected log log-evidence:
Ep(a) [ogg(()] 2 ((x;0,0)
(13)
Proof. First, notice that the expected log-evidence is equal to the negative entropy (X) = (p(=)) p(=)]. Given any random variable Z, the entropy can be decomposed into conditional entropy and mutual information terms: H (X) H(X Z) + I(X:
The expected log-evidence relates to the multimodal ELBO as follows:
Ep(w) ((x) ([X Z) 1(X; 2)
(14)
2 (= ax) [log (x x)] Ep(x)) [DL (po(z x)) [())]
(15)
[x:; 0, Φ)
(16)
where the inequality follows from the variational approximations of the respective terms. As in Alemi et al. (2017), we can use the following variational bounds.
For the conditional entropy, have -HXX| Z)
Ep(e))o(x æ) [logp(>) [z] [=]]
(17)
(p(x)pu(z m) g [=]] + Ep(=) (DK((p(a 2) g =))]
(18)
(p(e))po(= œ) go =)]
(19)
where ( x) is variational decoder that is parameterized by Φ. For the mutual information, we have
- -I(X; 2)
Ep(xe) [DiI. (po () [[>)]
(20)
Ep(cr) [DL (po [ x) || q[=)]] (1xx))
(21)
Ep(c)) [DiI (po (z () [| [[])]
(22)
where q(2) is prior.
Hence, the multimodal ELBO forms a variational lower bound on the expected log-evidence:
By( [log p(x)) = L(x; 0, Φ) + (v,
(23)
2 L [(x; 0 Φ)
(24)
where
(m, x) Ep(x) (p(o || go a x))] + (p(x) ([()
(25)
denotes the (non-negative) variational approximation gap.
14

---

Published as a conference paper at ICLR 2022
B.2 RELATION BETWEEN THE DIFFERENT OBJECTIVES
Proposition 2 relates the multimodal ELBO L from Definition to the objective Ls. which is general formulation of the objective maximized by all mixture-based multimodal VAEs. Compared to previous mixture-based formulations (Shi et al., 2019; Sutter et al., 2020), our formulation is more general in that allows for arbitrary subsets with non-uniform mixture coefficients. Further, the derivation quantifies the approximation gap between L and Ls. where the latter corresponds to the objectives that are actually being optimized in the implementations of the MMVAE, MoPoE-VAn VAE, and MVAE without sub-sau sub-sampling.
Proposition 2. For every stochastic encoder p (2 a:) that is consistent with Definition 2, the following inequality holds:
L(x; 6) 2
(26)
Proof. Recall the multimodal ELBO from Definition 1:
((:; 0 +) (x m) [lo (a =)] - Ep(a) [DKL (po x)) [1)))
(Dk
(27)
For the encoder po (= m²) plug in the mixture-based encoder p (= m) Eves WA po
from Definition 2 and re-write follows:
pp am) [log 9 (x =)] - Ep(n) (x (= x>) q[))]
(28)
=
Epla) Exes ua pol= m.) man |log go (x x)]
(29)
Ep(xx) E p5 x) - log [())]
= Σ WAl Bp(m))o(= | (x x)] - (p(r))o(= ma) |x)]+
(30)
AES
Bp(x)pa(= / [2) [=)]]
Σ wn E | x)]
<x). (=|x))]
(31)
ECtea (()))
Σ wA (=
=)] + Ep(ce) (Kk aa)) 11 |m))]
(32)
AES
Bp(a) (po xA) || [()))]}
2 Σ WA { log 2)] - Expce) (Pka ax)) (())))
(33)
AES
Ls ④)))
(34)
In Equation (31), CE(p, q) denotes the cross-entropy between distributions p and For Equation (32),
decompose both cross-entropy terms using CE(p, q) H(p) + DRL || g) and notice that the
respective entropy terms cancel out. The inequality (Equation (33)) follows from the non-negativity
of the KL-divergence. This concludes the proof that Ls( 0, Φ) forms lower bound on L(x; 0, Φ).
Objectives of individual models Sutter et al. (2021) already showed that Equation (28) subsumes the objectives of the MMVAE, MoPoE-VAE. and MVAE without ELBO sub- sub-sampling. However, in their actual implementation, all of these methods take the sum out of the KL-divergence term (e.g., see Shi et al., 2019, Equation 3), which corresponds to the objective L.s. To how L.s recovers the objectives of the individual models, simply plug in the model-specific definition of S into Equation (33) and use uniform mixture coefficients WA 1/|S| for all subsets. For the MVAE without ELBO sub-sampling
S is comprised of only one subset, the complete set of modalities (II..... .M} }. For the MMVAE, S is comprised of the set of unimodal subsets {æ.)..... { }} For the MoPoE-VAE. S is comprised of the powerset P(M) (0). Further implementation details, such as importance sampling and ELBO sub-sampling, are discussed in Appendix C.3.
15

---

Published as a conference paper at ICLR 2022
B.3 OBJECTIVE Ls SPECIAL CASE THE VIB
Lemma 1. Ls(x; 0, σ) is special case of the variational information bottleneck (VIB) objective
min Σ
WA (H 24
(35)
AES
where the encoding =
fe is a function of subsel XA, the terms H ZA) and (x: ZA) denote variational upper bounds of H(X Za) and [(X a: Za) respectively, and s summarizes the parameters of these variational estimators.
Proof. We start from Ls, the objective optimized by all mixture-bass multimodal VAEs. Recall from Definition
6s(x;0,0) Σ WA Ep(x)ra(= mn) log (x x)] Bp(r)) [DKL (po(z 2 xA) [[]])]]]]]] (36)
AES
(18)
Each term within the sum is comprised of two terms: (i) the log aig-likelihood estimation based on variational decoder 9 (x z): (it) the regularization of the stochastic encoder Po(z xx)) with respect to variational prior (==). The sampled encoding po(z aa can be viewed as the output of function 2 so (Xx,, a subset of modalities.
To see the relation to the underlying information terms H(X ZA) and T(X Za), we undo the variational approximation for (i) and (ii) by re-introducing the unobserved ground truth decoder p(æ| z) and the ground truth prior p(z).
For (f), we have
Ep(x))p((> aa) [log qo (x |z] S (p(a))a((= I aa) [log (a( =)] +
(37)
Eptz) [Dxc(((> 2) 1 x))]
Bp(x)pa(= m [logp(æ =)]
(38)
((( Za)
(39)
For (ti), we have
By(m) [poL æA) [[)))[ 2 Ep(a) [DL (po(z xA)) ll {())) D (p(z) (])())
(40)
= Ep(c)) [DiL (po(z @A) p()))]
(41)
I(X Z
(42)
Since Ls(x; 0, Φ) is being maximized, (i) is being maximized, while (ii) is being minimized. The maximization of (i) equal the minimization of variational upper bound on H(X Z Similarly, the minimization of (fi) is equal to the minimization of variational upper bound on T(XA; A; Z.). Hence, we have established that (æ 0, Φ) a special case of the more general VIB objective (Equation (35)) where the information terms are estimated with mixture-hased multimodal VAE that is parameterized by v (0.
B.4 DECOMPOSITION OF THE CONDITIONAL ENTROPY FOR SUBSETS OF MODALITIES
Lemma 2. Let X C X be some subset of modalites. If f(X where f is some function of the subset XA. then the following equality holds:
((X ZA) 11 (1.......MMM) MAA XA) + ZA)
(43)
Proof. When ZA is function of subset XA C X, have the Markov chain ZA t XA (I)....M)A since ZA is a function of the (observed) subset of modalities and depends the remaining (unobserved) modalities only through XA-
16

---

Published as a conference paper at ICLR 2022
We re-write [(X ZA) as follows:
[ 2 = H(X ZA, XA + AA)
(44)
H[X X T(X; (4a)
(45)
((
Xx) + +(x; X ZA)
(46)
= ((.. Xx) + H(XA| ZA)
(47)
Equation (44) applies the definition of the conditional mutual information. Equation (45) is based on
the conditional independence X _L ZAI XA implied by the Markov chain. Equation (46) removes
the "known" information that we condition on. Finally, Equation (47) follows from X X, which
implies that [(X; XA) H (Xxa) and XA ZA) = H (XA ZA).
B.5 PROOF OF THEOREM 1
Theorem 1. Each mixture-basel multimodal VAE (Definition 3) approximates the expected log-
evidence up to an irreducible discrepancy A(X, S) that depends on the model-specific mixture distribution S as well on the amount of modality-specife information in X.
For the maximization of Ls (0;;,0) and every value of 0 and o. the following inequality holds: Ep(a) [logp(x)) 2 Ls(x; 0 Φ) ((X,S)
(4)
where
A(X,S) EuA H (I... XA)
AES
In particular, the generative discrepancy always greater than or equal to zero and is independent of and é and thus remains constant during the optimization.
Proof. Lemma 1 shows that all mixture-basee multimodal VAEs approximate the expected log- evidence via the more general VIB objective
min
Σ WA (X ZA) + I (x)
(48)
AES
where the encoding ZA (x( A) is function of subset X C.
The fact that ZA is a function subset, permits the following decomposition of the conditional entropy (see Lemma 2):
H(X ZA) H( (I..... XA) H ZA)
(49)
In particular, Equation (49) holds for every ZA = f= and thus for every value tb Further, notice that H (I.....MM) (())....
Xx) is independent of the learned encoding Z and thus remains constant during the optimization with respect to v).
Hence, for every value vh, the following inequality holds:
( ZA) 2 H(X ZA)
(50)
2 (X X
(51)
which means that the minimization of H&(XX (ZA) is lower-bound by H (X(1.... MP.....MM44 X even
if H (X Za) is tight estimator of H(X) ZA))
Analogously, for the optimization of the VIB objective (Equation (48)), for every value v the following inequality holds:
Σ
WA H
[ ZA) - To (XA: ZA)
(52)
AES
2
Σ
wi [H(X) ZA) + T 1; Zx)]
(53)
AES
Σ
WA (H(XA) ZA) + (X ZA)1 + Σ
WA H(X (L.... XA)
(54)
AES
AES
A(X.S)
17

---

Published as a conference paper at ICLR 2022
where A S) independent of 1) and thus remains constant during the optimization. Consequently, ((X, represents an irreducible error for the optimization of the VIB objective.
For mixture-based multimodal VAEs, Lemma shows that Cs(æ; 0, Φ) is special case of the VIB objective with & = (0, Hence, for every value and @, the following incquality holds:
R (ogg)))gp(())) 2 (a; 0) + A(X
(55)
The exact value of ( depends the definition of the mixture distribution S, well the amount of mppelit variation in the data. In particular, A(X, > O, if there is any subset A 8 with WA for which H/ (1 (1.....AA XA)
B.6 PROOF OF COROLLARY
Corollary 1. Without modality sub-sampling, A(X,S) 0.
Proof. Without modality sub-sampling, is comprised of only one subset, the complete set of modalities (1,.... M). and therefore X X and X (1.....M = 0. It follows that A (
H ( MAA Xx) HQ) X) = O, since the conditional entropy of the empty set is zero.
B.7 PROOF OF COROLLARY 2
Corollary 2. For the MMVAE and MoPoE-VAE the generative discrepancy increases given an additional modality XM4 b. if the new modality is sufficiently diverse in the following sense:
Σ 1( (I... XM41 Xa)=
< S
Σ AES
H(X
[M44)) +
(56)
[s+
[S]
AES
1
H(XM+1 X)
(57)
S
where S denotes the model-specific mixture distribution over the set of subsets of modalities given modalities X1...., ... and S the respective mixture distribution over the extended set of subsets of modalities given X
Proof. Let XM+1 be the new modality, let X ( denote the extended set of modalities, and let S+ denote the new mixture distribution over subsets given X+, Note that all subsets from S still contained in St, but that S contains subsets in addition to those in S. Further, due to the re-weighting of mixture coefficients, S can have different mixture coefficients for the subsets shares with S. We denote by S ((A,wt) E S+ A S} the of new subsets and let w denote the new mixture coefficients, where typically WA * w due the re-weighting.
We are interested in the change of the generative discrepancy, when we add modality XM+11
A(X+ ST A(X,8)
(58)
= Σ W H(X (I..I...M
XB) Σ WA H(X (I.... XA).
(59)
BESI
AES
18

---

Published as a conference paper at ICLR 2022
Re- wr-tr the right hand side in terms of subsets that are contained in both S and S and subsets that are only contained in S* For this, we decompose the first term as follows
Σ BES
W H(X (I...,,,+IJ XB)
(60)
Σ
w H(X (1.... MI X
XA)+ Σ
WB (X (I...,M-I XB)
(61)
AES
BES-
Σ
w H MAA X
Σ
w HXM+11 X)+
(62)
AES
AES
Σ BES-
wB H (4100 XB)
(63)
where the last equation follows from
H H(X M+IAA XA) H M)AA
X + H(X++1 XA, X (A....AAA))))
(64)
(1...... XA) + (XM+1 X).
(65)
We can use the decomposition from Equation (63) to re-write the right hand side of Equation (59) by collecting the corresponding terms for H ( MHA Xa):
Σ
(wt WA)H (1, MA... XA) Σ
w HXM++ X)+
AES
AES
(66)
Σ BES-
wH MHNN (BB)
Notice that in Equation (66) only the first term can be negative, due to the re-weighting of mixture coefficients for terms that do contain X X Hence, in the general case, the generative discrepancy can only decrease, if the mixture coefficients change in such a way that the first term in Equation (66) dominates the other two terms.
For the relevant special case of uniform mixture weights, which applies to both the MMVAE and MoPoE-VAE. we can further decompose Equation (66) into (i) information shared between X and XM+1, and (it) information that is specific to X or X
Using uniform mixture coefficients шA [S| and w 18+1 for all subsets, we can factor out the coefficients and re- write Equation (66) follows:
1
Σ AES
H(X (1. (M. XA)
Σ
H(XM+1
X)
S+
S
|S+
AES
(67)
Σ
H(X
(1...,
XB)
S++
BES-
where the second term already denotes information that is specific to XM+1. Hence, we decompose the first and last terms corresponding to (i) and (ii).
For the first term from Equation (67), we have
H(X (1 MIA Xx)
(68)
S+
AES
-(13-)
1
(I... XMr1)) T( MAAs Xast1 Xa)}.
(69)
S
[S]
AES
For the last term from Equation (67), we have
Σ H(X
(1....M+IHVB XB)
(70)
S+
BES-
|st H(X|
XM+1)+ Σ
CIAUU(MFI)ESS H( (I.,, XA. XM4ta)
(71)
AES
19

---

Published as a conference paper at ICLR 2022
where we can further decompose
S+1 H(X
[M+1)= = S+1 H(X XA. + + |xM++)}
(72)
H(X XA: XM+1) H(x (XM+1)
(73)
[+|
H (I..,, XM+1) + H XM+1)}. (74)
S
AES
Collecting all corresponding terms from Equations (69), (71) and (74), we can re-write Equation (67)
follows:
1
Σ AES
(1
MYAA XA, XM41) +
(75)
S+
S
SS+||S|
I(X (1 (1... M)ai XA) +
(76)
|st|
[S]
AES
Σ
1(AALN-- (1... XM+1) +
(77)
S+
AES
Σ
H(x XM+1) +
(78)
S+ SS
Aes
H(X X).
(79)
S
AES
For both the MMVAE and MoPoE, the first and last terms cancel out, which can see by plugging in
the respective definitions of S into the above equation. Recall that for the MMVAE, S is comprised of the set of unimodal subsets {{11).... { and thus S+ is comprised
{x1h..... (+M+1) }- For the MoPoE- VAE, S is comprised of the powerset P(M) \ (0) and thus S is comprised of
the powerset ((M + 1) {Ø}. Hence, for the MMVAE and MoPoE- VAE, we have shown that
A X .S is equal to the following expression:
1
Σ AES
1(X (1 MM..MMAAM XAa)
(80)
S+
S
Σ AES
H(X A XM+1)
Σ AES
H(X X)
(81)
IS |S|||||
[s+
where the information is decomposed into:
(i) information shared between X and X (term (80),, and
(i)) information that is specific to X or XM+1 (the first and second terms in (81) respectively),
and where only (i) be negative since |S+| |S]. This concludes the proof of Corollary 2, showing that A(X++S*) A(X, > 0, if M+1 is sufficiently diverse in the sense that (fi) > (i).
20

---

Published as a conference paper at ICLR 2022
C EXPERIMENTS
C.I DESCRIPTION OF THE DATASETS
PolyMNIST The PolyMNIST dataset, introduced in Sutter et al. (2021), combines the MNIST dataset (LeCun et al., 1998) with crops from five different background images to create five synthetic image modalities. Each sample from the data is a set five MNIST images (with digits of the same class) overlayed on 28 x 28 crops from five different background images. Figure la shows 10 samples from the PolyMNIST dataset; each column represents one sample and each row represents one modality. The dataset provides convenient testbed for the evaluation of generative coherence, because by design only the digit information is shared between modalities.
Prol This new dataset is conceptually similar to PolyMNIST in that a digit label is shared between five synthetic image modalities. The difference is that in the creation of the dataset, we change the size and position of the digit, as shown in Figure 1b. Technically, instead of overlaying full-sized 28 x 28 MNIST digit on a patch from the respective background image, we downsample the MNIST digit by a factor of two and place it at random (1r y))-cordiinate within the 28 x 28 background patch. Conceptually, these transformations leave the shared information between modalities (i.e., the digit label) unaffected and only serve to make more difficult to predict the shared information across modalities on expectation.
Caltech Birds (CUB) The extended CUB dataset from Shi et al. (2019) is comprised of two modalities, images and captions. Each image from Caltech-Birds (CUB-200-2011 Wah al., 2011) is coupled with 10 crowdsourced descriptions of the respective bird. Figure 1c shows five samples from the dataset. It important to note that we use the CUB dataset with real images, instead of the simplified version based on precomputed ResNet-features that was used in Shi et al. (2019; 2021).
C.2 IMPLEMENTATION DETAILS
Our experiments are based on the publicly available code from Sutter et al. (2021), which already provides an implementation of PolyMNIST. A notable difference in our implementation is that we employ ResNet architectures, because we found that the previously used convolutional neural networks did not have sufficient capacity for the more complex datasets we use. For internal consistency, we use ResNets for PolyMNIST as well. We have verified that there is no significant difference compared to the results from Sutter et al. (2021) when we change to ResNets.
Hyperparameters All models were trained using the Adam optimizer (Kingma and Ba, 2015) with learning rate 5e-4 and a batch size of 256. For image modalities we estimate likelihoods using Laplace distributions and for captions we employ one-hot categorical distributions. Models were trained for 500, 1000, and 150 epochs on PolyMNIST, Translated- PolyMNIST, and CUB respectively. Similar to previous work, we use Gaussian priors and latent space with 512 dimensions for PolyMNIST and 64 dimensions for CUB. For a fair comparison, we reduce the latent dimensionality of unimodal VAES proportionally (wrt. the number of modalities) to control for capacity. For the 3-ablations, we use B E (3e-4, 3e-3, 3e-1, 1, 3, 9} and, in addition, 32 for CUB.
Srunslated-PolyMNISSS
Evaluation metrics For the evaluation of generative quality, we use the Fréchet inception distance (FID; Heusel et al., 2017), standard metric for evaluating the quality of generated images. In Appendix C.3, we also provide log-likelihoods and qualitative results for both images and captions. To compute generative coherence, we adopt the definitions from previous works (Shi al., 2019; Sutter et al., 2021). Generative coherence requires annotation on what is shared between modalities; for example, in both PolyMNIST and Translated-PolyMNV For single generated example 2. ~ { z) from modality m, the generative coherence is computed as the following indicator:
the digit label is shared by design.
Coherence(@m. ys gm.) 1 ((...)) =
(82)
where is ground truth class label and 9m is pretrained classifier (learned on the training data from modality m) that outputs a predicted class label. To compute the conditional coherence accuracy, average the coherence values over set of N conditionally generated examples, where N is
21

---

Published as a conference paper at ICLR 2022
typically the size of the test set. In particular, when Em (0(mm =) is conditionally generated from x ~ po( æA) such that A = {1...., ".I...,,M) M) \ m, the metric is specified as the leave-one-owe conditional coherence accuracy, because the input consists of all modalities except the one that is being generated. When it is clear from context which metric is used, we refer to the (leave-one- out) conditional coherence accuracy simply as generative coherence. For PolyMNIST, pretrained digit classifiers that are provided in the publicly available code from Sutter et al. (2021) and for Translated-PolyMNNSS we train the classifiers from scratch with the same architectures that are used for the VAE encoders. Notably, the new pretrained digit classifiers have a classification accuracy between 93.5-96, 96.9% on the test set of the respective modality, which means that it is possible to predict the digits fairly well with the given architectures.
the
C.3 ADDITIONAL EXPERIMENTAL RESULTS
Linear classification Shi et al. (2019) propose linear classification as a measure of latent fac- torization, to judge the quality of learned representations and assess how well the information decomposes into shared and modality-specific features. Figure 6 shows the linear classification accu- racy on the learned representations. The results suggest that not only does the generative coherence decline when we switch from PolyMNIST to Hounsaated-PlyMNNEEE
but also the quality of the learned representations. While a low classification accuracy does not imply that there is no digit information encoded in the latent representation (after all, digits show up in most self recosstuttuctc the result demonstrates that a linear classifier cannot extract the digit information.
Log-likelihoods and qualitative results Figure 7 shows the generative quality in terms of joint log-likellhoods. We observe a similar ranking of models as with FID, but we notice that the gap between MVAE and MoPoE-VAB appears less pronounced. The reason for this discrepancy is that, to be consistent with Sutter et al. (2021), we estimate joint log-likelihoone given all modalities- a procedure that resembles reconstruction more than does unconditional generation. It can be of independent interest that log- lielliiksssodss might overestimate the generative quality for unconditional generation for certain types of models. Qualitative results for unconditional generation (Figure 9) support the hypothesis that the presented log-likelihoods do not reflect the visible lack of generative quality for the MoPoE-VAF. Further, qualitative results for conditional generation (Figure 10) indicate lack of diversity for both the MMVAE and MoPoE-VAs VAE: even though draw different samples from the posterior, the respective conditionally generated samples (i.c., the ten samples along each column) show little diversity in terms of backgrounds or writing styles.
Repeated modalities To check if the generative quality gap is also present when modalities have similar modality-specifff variation, we use PolyMNIST with "repeated" modalities generated from the same background image (illustrated modalities from 2 to 5, but in contrast to the results from Figure 3, we now use repeated modalities. Figure 11 confirms that the generative quality of both the MVAE and MoPoE-Va. VAE deteriorates with each addi- tional modality, even in this simplified setting with repeated modalities. In comparison, the generative quality of the MVAE is much closer to the unimodal VAE for any number of modalities. These results lend further support to the theoretical statements from Corollaries 1 and 2.
Figure 5). We vary the number of
Figure 5: PolyMNIST with five "repeated" modalities.
MMVAE with the official implementation The empirical results of the MMVAE in Section 5 are based on simplified version of the model that was proposed by Shi et al. (2019). In particular, we use the re-implementation from Sutter et al. (2021), which optimizes the standard BLOO and not the doubly reparameterized ELBO gradient estimator (DReG, Tucker et al., 2019) with importance sam- pling that is used in the official implementation from Shi et al. (2019). Further, the re-implementation does not parameterize the prior but uses a fixed, standard normal prior instead.
To verify that these implementation differences do not affect the core results- generative quality gap and the lack of coherence we conducted experiments using the MMVAE with the official implementation from Shi et al. (2019). Figure 12 shows the 3-ablation for PolyMNIST and it confirms that there is still a clear gap in generative quality between the unimodal VAE and the MMVAE when we use the official implementation. For Translated-PolyMMISS (not shown) the
22

---

Published as a conference paper at ICLR 2022
results are similar; in particular, we have verified that generative coherence for cross generation is random, even we limit the dataset to two modalities.
MVAE with ELBO sub-sampling For the MVAE, Wu and Goodman (2018) introduce ELBO sub-sampling as an additional training strategy to learn the inference networks for different subsets of modalities. In our notation, ELBO sub-sampling can be described by the following objective:
C [(:; 0, 6) + Σ ((xx:),0) AES
(83)
where S denotes set of subsets of modalities. Wu and Goodman (2018) experiment with different choices for S, but throughout all their experiments they use at least the set of unimodal subsets {(z1),-... {mar} }, which yields the following objective:
((:; +) Σ (a,;; 0,
(84)
is important to note that the above objective differs from the objective optimized by all mixture- based multimodal VAEs (Definition 3) in that there are no cross-modal reconstructions in Equa- tion (84). As consequence, ELBO sub-sampling puts more weight on the approximation of the marginal distributions compared to the conditionals and therefore does not optimize proper bound on the joint distribution (Wu and Goodman, 2019).
Figure 13 shows the PolyMNIST 3-ablation comparing MVAE with and without ELBO sub-sampling. MVAR+ denotes the model with ELBOO sub-sampling using objective (84). Notably, MVAE+ achieves significantly better generative coherence, while both models perform similarly in terms of generative quality (both in terms of FID and joint log-likelihood). Hence, even though the MVAE optimizes incorrect bound the joint distribution (Wu and Goodman, 2019), results suggest that the learned models behave quite similar in practice, which can be independent interest for future work.
Linear Classification Accuracy
1.0
Linear Classification Accuracy
1.0
MVAE MMVAE
0.8
0.8
MoPoE-VAE
0.6
0.6
MVAE
0.2-
MMVAE
MoPoE-VAE
0.0
10P 11F 0.0
11
10"2
111
10P
8
(a) POLYM
(b) Translated-Poon PranslatedPolyMMNIST
Figure 6: Linear classification of latent representations. For each model, linear classifiers were trained on the joint embeddings from 500 randomly sampled training examples. Points denote the average digit classification accuracy of the respective classifiers. The results are averaged over three seeds and the bands show one standard deviation respectively. Due to numerical instabilities, the MVAE could not be trained with larger 33 values. For CUB, classification performance cannot be computed, because shared factors are not annotated.
23

---

Published as a conference paper at ICLR 2022
9030
3000
-160
Log- Pag-Likllhood
Likelihood
Log- Likellhood
1003
-5300
-800
-163
MMVAE MoPcE VNC
MMVAE MaPoC VNC
4403
MMVAA
MURLE-VAE
10030
00000
VAE
-900
22
00
10
1311
10
12:2
10
16
0
(a) PolyMNIST
(b) Translated-PolyMNNE
(c) Caltech Birds (CUB)
Figure 7: Joint log-likelihon likelihoods over a range of ß values. Each point denotes the estimated joint log- likelihood averaged over three different seeds and the bands show standard deviation respectively. Due to numerical instabilities, the MVAE could not be trained with larger / values.
191
(a) X1
(b) X₂
(C) X3
(d) X
(e) Xs.
Figure 8: FID for modalities Xi...., X3. The top shows all FIDs for PolyMNIST and the bottom row for Translated-PolyMNNITT respectively. Points denote the FID averaged over three seeds and bands show one standard deviation respectively. Due numerical instabilities. the MVAE could not be trained with larger 3 values.
24

---

Published as a conference paper at ICLR 2022
(al unimodal VAE, =
(b) MVAE, & =1
(C) MMVAE, /
(d) MoPoE- VAE, 3
(c) unimodal VAE, 0.3
O MVAE, 8 0.3
( MMVAE, 3 0.3
(h) McPoE-VAE, 0.3
unimodal VAE,
MVAE,
(3) MMVAE,
McPoE-VAL VAE,
your 1E9
(m) unimodal VAE, ,
MVAE,
(0) MMVAE, /
(p) VAEE
Figure 9: Qualitative results for the unconditional generation using prior samples. For PolyMNIST (Subfigures (a) to (d)) and HuunslatedPolyMNNIs (Subfigures (e) (h)), show 20 samples for each modality. For CUB, we show 100 generated images (Subfigures (i) to (I)) and 100 generated captions (Subfigures (m) (p)) respectively. Best viewed zoomed and in color.
25

---

Published as a conference paper at ICLR 2022
5
5
6
8
6
2
6
8
(a) MVAE, B
(b) MMVAE, 8
(c) MoPoE VAE, /
(d) MVAE, / 0.3
(c) MMVAE, 0.3
(6) MAPEE-VAE, 0.3
MVAE, 9 9.0
(b) MMVAE, 9.U
MoPot VAE, / 9.0
① MVAE, 9.0
(3) MMVAE, 8 9.0
(1) MoPol. VAE, 9.0
Figure 10: Qualitative results for the conditional generation across modalities. For PolyMNIST (Subfigures (a) (c)) and Translated-PolyyMe PolyMNIST (Subfigures (d) (f)), show 10 conditionally generated samples of modality X1 given the sample from modality X2 that is shown in the first row of the respective subfigure. For CUB, show the generation of images given captions (Subfigures (g) to (i)), as well as the generation of captions given images (Subfigures (j) to (I)). Best viewed zoomed and in color.
26

---

Published as a conference paper at ICLR 2022
MVAE MMVAE
MoPoE-VAE unimodal VAE
175
MVAE
MoPoE-VAE
MMVAE
unimodal VAE
150
150
125
FD
75-
50-
50-
23
number of modalities
number of modalities
(a) PolyMNIST
(b) Translated- Translatcd-PolyyMNIST
Figure 11: Generative quality a function of the number of modalities. In contrast Figure 3, here we "repeat" the same modality, to verify that the generative quality also declines when the modality- specific variation of all modalities is similar. All models trained with = 0.3 on MaanslatddPolyyMyy bands show one standard deviation respectively. For the unimodal VAE, which uses only a single modality, the average and standard deviation are plotted as a constant.
PolyMNIST and The results are averaged over three seeds and all modalities; the
5000
300
MMVAE
unimodal VAE
6000
Log- Log-likelihood
200
70000
-8000
50
-9000
MMVAE unimodal VAE
10P
10 -10000
10
10'
3
(a) FID
(b) Joint lieeliied
Figure 12: PolyMNIST 3-ablation using the official implementation of the MMVAE. In particular, for both the MMVAE and the unimodal VAE, we use the DReG objective, importance sampling, as well learned prior. Points denote the value of the respective metric averaged over three seeds and bands show one standard deviation respectively.
174
Condiiooal Cohorenca Accuracy
MVAE
MUAE MVAE*
MvAE
MVNE
MVAE!
Log: Likellrood
-6000
200
-7000
00
-7000
10'55
12
10-1
10-2
(a) FID
(b) Joint log- lieelikllhood
(c) Generative coherence
Figure 13: PolyMNIST 3-ablation, comparing MVAE with and without additional ELBO sub- sampling. MVAE denotes the model with additional ELBO sub-sampling. Points denote the value of the respective metric averaged over three seeds and bands show one standard deviation respectively.
27


Reviews:
[{"review_id": "iXT6lMxhEH9", "reviewer": "Reviewer_B728", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is clearly written and easy to follow. The topic of this paper is very interesting and could have some implications in the area of VAE, or more broadly, generative models. \n\nThe strengths of this paper include the following.\n- The writing in most parts of the paper is clear. \n- The theory presented in section 4 is concise but very informative. \n- The experiments are extensive.\n\nThe weakness of this paper includes the following. \n- As an addition to Thm 1, it is useful to give an example that compares the ELBO and the gap $\\Delta$. This should tell us which term dominates. If the ELBO dominates then your theory may not be able to explain the quality drop.\n- As for the claim that the gap $\\Delta$ degrades quality of multimodal VAEs compared to unimodal VAEs, I think the authors should have a paragraph discussing the following. Are you assuming that a larger gap $\\Delta$ leads to worse quality? However, optimizing the ELBO can sometimes be better than optimizing the likelihood (e.g. VAE sometimes outperforms likelihood-based models such as normalizing flows), especially when the model has limited capacity - as in this case, likelihood-based models suffer from memorizing a huge amount of modes. From this view, the gap between likelihood and ELBO (for unimodal VAE) may have good inductive bias. So, could the $\\Delta$ in Thm 1 have any good inductive bias? \n- Following the above point, it is also valuable to conduct an experiment where you gradually increase $\\Delta$ and see if the quality monotonically drops. \n- There is no theory for coherence. In section 5.2, the description about this concept is not very clear. What does it mean by saying \"a model's ability to generate semantically related samples across modalities\"? Also, a formal, mathematical definition for the leave-one-out coherence also helps understanding of this part. \n\nSome minors:\n- Eq (1): $\\sum_{A\\in\\mathcal{S}}$ is not precise because you are in fact summing over $(A,\\omega_A)$.\n- Figures should be made larger for better readability.\n\nThe evaluation of this paper is based on the above points. The score may be modified based on feedback and updates from authors.", "summary_of_the_paper": "The paper looks at the ELBO loss function for multimodal VAE, and theoretically proves a nontrivial gap $\\Delta(X,S)$ between the likelihood and ELBO, from an information theory point of view. The paper claims this gap leads to quality drop of multimodal VAEs. The paper then conducts experiments and empirically examines the tradeoff between unimodal and multimodal VAEs. ", "summary_of_the_review": "Theory is concise but informative, and experiments are extensive. However, there are a few key things missed from the paper, as discussed in the main review. \n\n--------------------\nUpdate after rebuttal\n\nI have read the response and I will keep my score. It is a very interesting paper. I encourage the authors to add some of the discussions to either the main text or the appendix. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "V5Jx-dZXBEe", "reviewer": "Reviewer_wNh5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n* The authors do a great job covering a wide body of research on multi-modal VAEs.\n* The paper proposes a theoretically grounded explanation for the empirically observed lower generation quality of multi-modal VAEs. The explanation appears sound.\n* Extensive experiments on simple datasets are carried out to further support the results.\n* Limitations of multi-modal VAEs discussed in the paper would be of interest of a wide audience beyond machine learning research.\n\n**Weaknesses**\n* The paper would benefit from a more detailed introduction of the datasets, especially PolyMNIST, e.g. why does it have 5 modalities and what are they?\n* I was not fully convinced by the comparison between VAEs that rely on sub-sampling and those that do not. In particular, VAEs that rely on sub-sampling see less data during training compared to those that do not. Would conclusions change if VAEs that rely on sub-sampling would be trained proportionally longer?", "summary_of_the_paper": "The paper discusses a surprising failure mode of multi-modal VAEs, in which their generation quality lags behind that of unimodal VAE. A theoretically grounded explanation based on the ELBO of the multi-modal VAE is proposed to explain this gap, and extensive ablation experiments validating implications of the proposed explanation are carried out. Finally, the theory and experiments proposed in the paper have implications for a large body of research into multi-modal VAEs.", "summary_of_the_review": "Fundamental limitations of multi-modal VAEs discussed and experimentally validated in the paper would be of interest of a large audience and can potentially impact many applications of these methods (e.g. in natural sciences, bioinformatics, etc). Intuition and theoretical justification of the limitations appear sound, and is supported by extensive experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UBgp7KjOVAY", "reviewer": "Reviewer_UGYa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is very well organized and easy to read.\n- It theoretically shows a fundamental problem in mixture-based multimodal VAEs, which has not been pointed out before. This problem is intuitively convincing.\n- The experiments using more complex datasets clearly support the theoretical results of this study. These results are clearly novel and important for future research in multimodal VAEs.\n\nWeaknesses:\n- The authors define models with Equation 2 as the objective as mixture-based multimodal VAEs, and they show the limitations of Equation 2 due to sub-sampling. However, this objective is a lower bound of the \"true\" objectives of MMVAE and MoPoE-VAE, as pointed out by the authors. Therefore, it is unclear whether the problems pointed out in this paper occur only in Equation 2 or also in these true objectives (I understand that the actual implementation in [Sutter+ 2021] uses Equation 2, but my point is that the authors should clarify the scope of this problem.).\n- The use of the term \"sub-sampling\" in this paper is ambiguous. In this paper, modality sub-sampling is a very important concept, and the authors show that it increases the irreducible discrepancy and makes the lower bound less tight. Sub-sampling here is supposed to mean extracting a subset from multiple subsets of modalities, but the authors do not define it clearly. Moreover, they also use the term to mean ELBO sub-sampling in MVAE, which may cause confusion. The authors should clarify the meaning of this term.\n\nMinor comment:\n- Is there any reason why the usual notation for encoders and decoders in VAEs ($p_{\\theta}$ and $q_{\\phi}$) are reversed in this paper? (Of course, there is nothing wrong with this, as it is consistent throughout the paper.)", "summary_of_the_paper": "In this paper, the authors show theoretically that in mixture-based multimodal VAEs, sub-sampling of modalities can make ELBO less tight. They also introduced a more complex dataset, Translated-PolyMNIST and Caltech Birds with real images, to reveal the limitations of multimodal VAEs in practice.", "summary_of_the_review": "Although some concerns remain about the terminology and the scope of the issue the authors are advocating, the paper makes some novel points and contributions to the study of multimodal VAEs. Therefore, I judge this paper to be above the acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "ODNS4MgF7Po", "reviewer": "Reviewer_tUe8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Multi-modal data analysis is an important topic with several real-world applications. The authors' contribution is to characterize the limitations of existing VAEs as a step towards improving these methods, which is quite valuable. However, for this conference, I expect the authors to also suggest a solution for the discussed limitations. While it is quite useful to quantify the generative model performance in the presence of multi-modal data, it is not surprising to see that when all modalities are present, the joint distribution approximation for VAEs that use subsets of modalities (MMVAE) is not as accurate as VAEs that use the joint posterior as a product of unimodal posteriors (MVAE). These are discussed by Sutter et al., ICLR2021 as well.\n\nThe paper is very well written and well organized. Both theoretical and experimental results seem valuable.  However, it would be nice if the authors could show how we can overcome the lack of generative quality due to the discrepancy, $\\Delta(x,s)$ in Theorem 1 and how we should balance the “sufficient diversity” and  “redundant information” with respect to the existing modalities. I think these are critical points that need to be somehow addressed.\n", "summary_of_the_paper": "The manuscript attempts to study some of the existing multi-modal VAE models, MVAE, MMVAE, and MoPoEVAE and to identify their limitations in terms of generative quality and generative coherence. The authors study the impact of sub-sampling on the generative quality and show that if the shared information among modalities cannot be fully predicted across modalities, the model would lack generative coherence. They demonstrate their findings for two multi-modal datasets, i.e., PolyMNIST and Caltech Birds. To better show the limitations of multi-modal VAEs when applied to more complex data, the authors made an augmented version of PolyMNIST, called Translated-PolyMNIST including 5 modalities. ", "summary_of_the_review": "The authors' contribution on formulating and quantifying the limitation of the existing VAEs for multi-modal data analysis is interesting and valuable. However, I think the authors need to at least address a few of these limitations to show how the current methods can be improved. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides an investigation into the quality of generations made by multimodal VAEs. All reviewers were in favor of accepting the paper, and there was quite a bit of detailed discussion and clarifications in the revised version of the paper which led two reviewers to raise their ratings. Overall this is an interesting contribution to the area and is an excellent fit for ICLR.
Decision: Accept (Poster)