Generalized Multimodal ELBO
Authors: Thomas M. Sutter
Imant Daunhawer
Julia E Vogt
Venue: ICLR 2021 Poster (2021)
Keywords: Multimodal
VAE
ELBO
self-supervised
generative learning

Abstract: Multiple data types naturally co-occur when describing real-world phenomena and learning from them is a long-standing goal in machine learning research. However, existing self-supervised generative models approximating an ELBO are not able to fulfill all desired requirements of multimodal models: their posterior approximation functions lead to a trade-off between the semantic coherence and the ability to learn the joint data distribution. We propose a new, generalized ELBO formulation for multimodal data that overcomes these limitations. The new objective encompasses two previous methods as special cases and combines their benefits without compromises. In extensive experiments, we demonstrate the advantage of the proposed method compared to state-of-the-art models in self-supervised, generative learning tasks.

Published as a conference paper at ICLR 2021
GENERALIZED MULTIMODAL ELBO
Thomas M. Sutter*
mann Daunhawer"
Julia E. Vogt
Department of Computer Science
ETH Zurich
8092 Zurich, Switzerland fthomas sutter, imant. daunhawer, vuu...ooohhheaaf.eeeee......
ABSTRACT
Multiple data types naturally co-occur when describing real-world phenomena and learning from them is long-standing goal in machine learning research. How- ever, existing self-supervised generative models approximating an ELBO are not able fulfill all desired requirements of multimodal models: their posterior ap- proximation functions lead to trade-off between the semantic coherence and the ability to learn the joint data distribution. We propose new, generalized ELBO formulation for multimodal data that overcomes these limitations. The new objective encompasses two previous methods as special cases and combines their benefits without compromises. In extensive experiments, we demonstrate the advantage of the proposed method compared to state-of-the-ar models in self- supervised, generative learning tasks.
INTRODUCTION
The availability of multiple data types provides rich source of information and holds promise for learning representations that generalize well across multiple modalities (Baltrušaitis et al.) 2018). Multimodal data naturally grants additional self-supervision in the form of shared information necting the different data types. Further, the understanding of different modalities and the interplay between data types are non-trivial research questions and long-standing goals in machine learning research. While super-su approaches have been applied successfully (Karpathy & Fei-Fei 2015 Tsai et al. 2019 Pham et al.) [019 Paboennaer-Seebar et al., 2019). the labeling multiple data types remains time consuming and expensive. Therefore, requires models that efficiently learn from multiple data types in a self-supervised fashion.
Self-supervised, generative models are suitable for learning the joint distribution of multiple data types without supervision. We focus on VAEs (Kingma & Welling 2014 Rezende et al. 2014) which are able to jointly infer representations and generate cess on unimodal datasets, there are additional challenges associated with multimodal data (Suzuki et al., 2016, Vedantam et al. 2018). In particular, multimodal generative models need represent both moodality-specifff and shared factors and generate semantically coherent samples across modal- ities. Semantically coherent samples are connected by the information which is shared between data types (Shi et al. 2019). These requirements are not inherent to the objective-the evidence lower bound (ELBO) of unimodal VAEs. Hence, adaptions the original formulation are required to cater to and benefit from multiple data types. Furthermore, to handle missing modalities, there is scalability issue in terms of the number of modalities: naively, it requires nd different encoders to handle all combinations for M data types. Thus, we restrict our search for an improved multimodal ELBO to the class of scalable multimodal VAEs.
observations. Despite their suc-
Among the class of scalable multimodal VAES, there are two dominant strains of models, based on either the multimodal variational autoencoder (MVAF, Wu Goodman 2018) or the Mixture-of- Experts multimodal variational autoencoder (MMVAF, Shi et al., 2019). However, we show that these approaches differ merely in their choice of joint posterior approximation functions. We draw theoretical connection between these models, showing that they can be subsumed under the class
"qqual contribution.

---

Published as a conference paper at ICLR 2021
of abstract mean functions for modeling the joint posterior. This insight has practical implications, because the choice of mean function directly influences the properties of a model (Nielsen 2019). The MVAE uses a geometric mean, which enables learning sharp posterior, resulting in good approximation of the joint distribution, On the other hand, the MMVAF, applies an arithmetic mean which allows better learning of the unimodal and pairwise conditional distributions. We generalize these approaches and introduce the Mixture-of-Produee Products-of-Experts-- VAE that combines the benefits of both methods without considerable trade-off's.
In summary, we derive a generalized multimodal ELBO formulation that connects and generalizes two previous approaches. The proposed method, termed MoPoE-VAL models the joint posterior approximation as a Mixture-of- oraoooo--roouutooo----a---- Efpexesss which encompasses the MVAE (Product-of- Experts) and MMVAE (Mixture-of -fixpere-o--Experrt) as special cases (Section 3} In contrast to previous models, the proposed model approximates the joint posterior for all subsets of modalities, an advan- tage that we validate empirically in Section where our model achieves state-of-the-art results.
RELATED WORK
This work extends and generalizes existing work in self-supervised multimodal generative models that are scalable in the number of modalities. Scalable in the sense that a single model approximates the joint distribution over all modalities (including all marginal and conditional distributions) instead of requiring individual models for every subset of modalities (e.g., Hu.ng et al. 2018 Tian & Engell 2019 Hsu & Glass 2018). The latter approach requires prohibitive number of models, exponential in number of modalities.
Multimodal VAFs Among multimodal generative models, multimodal VAEs (uzuki et al. Vedantam et al. 2018 Kurle et al., 2019 Tsai et al., 2019; Wu & Goodman 2018 Shi et al. 2020 Sutter et al. 2020) have recently been the dominant approach. Multimodal VAES are not only suitable to learn joint distribution over multiple modalities, but also enable joint inference given a subset of modalities. However, to approximate the joint posterior for all subsets modalities efficiently, it is required to introduce additional assumptions overcome the issue of scalability, previous work relies on either the product (Kurle et al., 2019) Wu & Goodman| 2018) the mixture (Shi et al., 2019 2020 of unimodal posteriors. While both approaches have their merits, there are also disadvantages associated with them. We unite these approaches in a generalized formulation-a mixture of products joint posterior-thal that encapsulates both approaches and combines their benefits without significant trade-offs.
2016
2019
the form of the joint posterior. To
Multimodal posteriors The MVAE (Wu & Goodman, 2018) assumes that the joint posterior is a product of unimodal posteriors- Product-of-Experr (PoE, Hinton 2002). The PoE has the ben- efit of aggregating information across any subset of unimodal posteriors and therefore provides an efficient way of dealing with missing modalities for specific types of unimodal posteriors (c.g.. Gaus- sians). However, to handle missing modalities the MVAE relies on an additional sub-sampling of unimodal log-likelihoods, which no longer guarantees valid lower bound on the joint log- ligeeliodo (Wu & Goodman 2019). Previous work provides empirical results that exhibit the shortcomings of the MVAE, attributing them to a precision miscalibration of experts (Shi et al. 2019) or to the av- eraging over inseparable individual belief's (Kurle et al.) 2019). Our results suggest that the PoE works well in practice, if it is also applied on all subsets of modalities, which naturally leads to the proposed Mixture-of of Prad--rooctsofffoo of -xperrs (MoPoE) generalization, which yields a valid lower bound on the joint log-likelibne likeliiiddo
On the other hand, the MMVAE (Shi et al. 2019) assumes that the joint posterior is a mixture of uni- modal posteriors-a a Mfxture-of- Expprrsss (MoE). The MMVAE is suitable for the approximation of unimodal posteriors and for translation between pairs of modalities, however, it cannot take advan- tage of multiple modalities being present, because it only takes the unimodal posteriors into account during training. In contrast, the proposed MoPoE- computes the joint posterior for all subsets of modalities and therefore enables efficient many-to to-mnyy translations. Extensions of the MVAE and MMVAE (Kurle et al.) 2019 Daunhawer et al. 2020 Shi et al.) 2020 Sutter et al., 2020 have introduced additional loss terms, however, these are also applicable to and can be added on top of the proposed model.
2

---

Published as a conference paper at ICLR 2021
Table I: Properties of previous scalable multimodal VAEs and our proposed model. Note that to deal with missing modalities, the MVAE requires sub-sampling of unimodal ELBOs, which yields an invalid bound on the joint log- likelihood (Wu & Goodman 2019).
Model
Posterior form
Aggregate modalities
Multi-modal posterior
Missing modalities
MVAE
PoE
MMVAE
MoE
X
MoPoE- VAE (ours)
MoPoE
Table 1 summarizes the properties of previous multimodal VAEs and highlights the benefits of the proposed model: the ability to aggregate multiple modalities, to leam multi-modal posterior (in the statistical sense), and to efficiently handle missing modalities at test time.
3 METHOD
3.1 PRELIMINARIES
We consider dataset (XO) of N ii.d. samples, each of which is set of M modalities x(O = { (M 1==1- We assume that the data is generated by some random process involving a joint hidden random variable z such that inter-modality dependencies unknown. The marginal log-likelihood can be decomposed into a sum over marginal log-likelihoode of individual sets
log (x)) Σim log (x(()), which can be written as:
log (x) DKL (9o X (Ipo [Ixx + L(0, x()),,)
with C(O, x : (x [log po (X (0)
[2]] DKL (1 X (lpo(())
(2)
[(,, 6; ()) is called evidence lower bound (ELBO) on the marginal log-likel of the i-th set. It forms a tractable objective to approximate the joint data distribution log (x( (z (x) the posterior approximation distribution with learnable parameters ó. From the non non- nog-neeativv
of the KI. divergence, it follows that log Ps (x(()) L(0, o; X If the posterior approximation
(x) is identical to the true posterior distribution po (x the bound holds with equality. Hence, maximizing the ELBO in Equation (2) minimizes the otherwise intractable KL-divergence
between approximate and true posterior distribution:
arg min DKL (go (=|x(()) Ipp (1x
(3)
Adaptations to the ELBO formulation in Equation 2 include additional hyperparameter / which weights the KL-divergence relative to the log-likes log-likelihood (Higgins et al. (017). To improve read- ability, we will omit the superscript (i) in the remaining part of this work.
3.2 APPROXIMATING Po(=[X) IN CASE OF MISSING DATA TYPES
For a dataset of M modalities, there are M different subsets contained in the powerset PXX). If, for particular observation, we only have access to a subset of data types X E P(X), the approximation of po (X&) would result in different ELBO formulation [(6, Φı; X&) where the true posterior Po x/Xx)) of subset X₂ is approximated. Instead, we are interested the true posterior po(2[%) of all data types X, even when only subset Xx, i.e. do, (z (x), is available. The desired ELBO for the available subset X5 is given by
La (0, ou: X) = Eg,, (x/Xe) [log(po
DKL (āo. (z (Pp))))
(4)
The subtle but important difference between L. (f, dx; X) and L(O, ou X5) is that the former still yields a valid lower bound on po (X), whereas the latter forms a lower bound on log po (X&), which no longer valid bound the desired log po (X)).
3

---

Published as a conference paper at ICLR 2021
Different from previous work, argue for optimization of the powerset P(X), i... the joint optimization of all ELBOs (b, σx; X) defined by the posterior subset approximation go, xxx)) Since maximizing the ELBO in Equation (2) is equivalent to minimizing the KL-divergence in Equa- tion (3), the joint optimization of the powerset P(X) is equal to the minimization of the following convex combination of Kl--divergences of the power set (XX)
arg min
E X DKL (ã x (xx))))())))
(5)
Hence, we propose to optimize Equation 4 for all subsets Xx.
Lemma 1. The sum of Kl.-divergences in Equation
describes the joint probability log po (X)) as
follows:
log (X))
1
Σ
DEL
E (x)x)( XEEP(X)
Σ
log Po 2 (o(z))
2
2
dol (x)
Following Lemmal T (see Appendix A. for the proof) and the non-negativity the KL-divergence, that the convex combination of expectations over the powerset P(X) is an ELBO the joint probability log Pa (X). Since this would require 2 different inference networks a naive implementation, we use more efficient approach utilizing abstract mean functions.
3.3 SCALABLE INFERENCE USING ABSTRACT MEAN FUNCTIONS
To create model that is scalable in the number of modalities model that breaks the need for 2M different networks- petworrs-previouu works define the joint posterior approximation (o(x|X) as a mean function of the unimodal variational posteriors. The PoE and MoE can be subsumed under the concept of abstract means (Nielsen 2019). Abstract means unify multiple mean functions My y for given function f (Niculescu & Persson 2005):
P Σ k=1
M
f(ps)
P
where is the number of elements and the function f needs to be injective in order for f-1 to exist. f(p) = aa results in the arithmetic mean, f(p) = log p in the geometric mean.
The choice of mean function directly influences the properties of the learned model as we will recapitulate with regard to multimodal VAEs in the following. The MVAE (Wu & Goodman 2018 employs the PoE, which is geometric mean of unimodal posteriors. Aggregation through the PoE results in sharp posterior approximation (Hinton 2002), but struggles in optimizing the individual experts as mentioned by the authors (Wu & Goodman 2018 3). In contrast, the MMVAE (Shi al. 2019) uses the MoE, which is an arithmetic mean unimodal posteriors. As such, the MMVAE optimizes individual experts well, but is not able to learn distribution that is sharper than any of its experts. Thus, the choice of mean function directly influences the properties of the resulting model. The MoE is optimizing for conditional distributions based the unimodal posterior approximations, while the PoE is optimizing for the approximation of the joint probability distribution.
For scalable, abstract-mean based models, the set of parameters du for the posterior approxima- tion of subset {z (X4) is determined by the unimodal posterior approximations 96, (=|æ,, as Φ (o j E (I....,Ad M. æz E }
3.4 GENERALIZED MULTIMODAL ELBO
In the following, first introduce the new ELBO (M (f; X. and then prove that its objective minimizes the convex combination of KL-divergences in Equation (
We omit the subscript & for the parameterization of the posterior approximations when it is clear from context that only Xg is available, and write o instead.
4

---

Published as a conference paper at ICLR 2021
Definition 1.
1. Let the posterior approximation of subset X be
Ts(z [x(=(x+)
V E xx)) & Imaexx do, (z(az)
2. Let the joint posterior be g0(z|x)) 2d ExLEP(x) {z(2)(x))
The objective LMoroE(0, o; X) for learning a joint distribution of multiple data rypes defined as LMoPoE (0, 00 X) == (x (log(pe DKL.
Σ
d (o(2)%))lloo(2))
(6)
2M
XxEP(2)
From Definition 1 Lemma directly follows.
Lemma 2. (M x) is multimodal ELBO, that is log (x 2 (MoPOE o; X).
Since (=((=(X) is defined as a mixture distribution (i.e., a probability distribution), it directly follows that MoPor (0, Φ; X) is a valid ELBO on log po (X). because variational distribution can be chosen arbitrarily as long as it is a valid probability distribution. For proof of Lemma see Appendix A.2
Lemma 3. Maximizing (MoPoE (0, o: X) minimizes the convex combination KL-divergences of the powerset (()) given in Equation
For proof of Lemma see Appendix A.3 Definition I does not put any restrictions the choice of posterior approximations Zo (2 &x).. As we are interested scalable, multimodal models, we focus methods which apply to this restriction and choose the PoE for the posterior approximations of the subsets X E PXX). Other, non-scalable posterior fusion methods are possible using this framework.
3.5 THE GENERAL FRAMEWORK
Definition 1 can be interpreted hierarchical distribution: first the unimodal posterior approxima- tions of subset qa, 2 V æ; X5 are combined using a PoE, second the subset approximations (z xx) vX, E PXX) are combined using MoE. This allows to combine the strengths of both MoE as well as PoE while circumventing their weaknesses (see Section 2). For Gaussian posterior approximations, is common in VAEs, the PoE can be calculated in closed form, which makes it computationally efficient solution.
In the following, derive the objectives optimized by the MVAE and MMVAE special cases of LOOPGE(0 Φ; X). The MVAE only takes into account the full subset, i.e., the PoE of all data types. Trivially, this MoE with only single component:
(ooE d: X)
DKL(
with (=(x x II goy (o, (2|m,) (2 Σ PaF((us,(E)1
(8)
This is equivalent the MoPoE- VAE of single subset Xx, which is the full set X.
As the PoE of single expert is just the expert itself, the MMVAE model (Shi et al. 2019) is the special case of EMoPOE (9, d: X) which takes only into account the M unimodal subsets:
M Σ f=1
(0, = (x [x1z) - DKL
go (z IIo (z)
(9)
M
M
1 M
with (x(x = M
Σ
To, (.)))
Σ
PoE (26,((
(10)
M
j=1
j=1
5

---

Published as a conference paper at ICLR 2021
LMoE o; X) is equivalent to MoPoE-VAL of the M unimodal posterior approximations 4o, (z) for M..
Therefore, the proposed MoPoB-VAL is generalized formulation of the MVAE and MMVAE, which accounts for all subsets of modalities. The identified special cases offer a new perspective on the strengths and weaknesses of prior work: previous models focus specific subset of posteriors, which might lead to a decreased performance on the remaining subsets.
In partieular, the MVAE should perform best when all modalities present, whereas the MMVAE should be most suitable when only a single modality is observed. We validate this observation empirically in Section
EXPERIMENTS & RESULTS
We evaluate the proposed method three different datasets and compare it to state-of-the-ard meth-
ods. We introduce a new dataset called PolyMNIST with 5 simplified modalities. Additionally, we evaluate all models on the trimodal matching digits dataset MNIST-SVHN-Tew and the challenging bimodal Celeba dataset with images and text. The latter two were introduced in Sutter et al. 2020)
We evaluate the models according three different metrics. We assess the quality of the learned latent representation using linear classifier. The coherence of generated samples is evaluated using pre-trained classifiers. The approximation of the joint data distribution is measured using test set log-likelihoods.
The datasets and the evaluation of experiments are described detail in Appendix
4.1 MNIST-SVHN-TE
Based on the MNIST-SVHN dataset (Shi ct al. 2019), this trimodal dataset with an additional text modal- ity forces model to adapt to multiple data types. It involves data types of various difficulties. Whereas
MOPOE
Coherence Rccurncy
MNIST (LeCun & Cortes 20T0) and text are clean modalities, SVHN (Netzer et al. 2011 is comprised of noisy images.
Tables 2 and 3 show the superior performance of the proposed method compared state-of-the-ant methods regarding the ability to learn meaningful latent rep- resentations and generate coherent samples. MVAE reaches superior performance for the generation of the SVHN modality, while MoPoE-VAE overall achieves best coherence results. Table 4 shows the results for the test log- ligeliikis...s.
-166
-2008
-lice
-1590
Log Liglibiliaaae
Figure 1: Joint Coherence vs. Log- Likelihoods for MNIST-SVHN-Tex.
The proposed MoPoE-VALE is the only method that is able to reach state-of-the-art coherence, latent classification accuracies, well as test ligliiiels for all combination of inputs. This can be seen Figure [ illustrating the trade-off between test log- ligeliikss and joint coherence for every model. Every point encodes the joint coherence and joint logeliield for different 3- value The goal is to have high coherence and log-likelihoods (i.e., the top right corner). Note that lower beta values typically correspond models with higher log-likelihood but lower coherence. Overall, the MoPoE- VAE achieves superior trade-off compared to the baselines. As expected by our theoretical analysis (Section 3.5). the MVAE achieves good joint log likeliosssosss whereas MMVAE reaches high joint coherence.
The
"The j-hyperparameter controls the weight of the Kl.-divergence in Equation 6 We evaluate the models using E (0.5, 1.0, 5.0, 10.0, 20.0).

---

Published as a conference paper at ICLR 2021
Table 2: Linear classification accuracy of latent representations for MNIST-SVHN-Tex. We eval- uate all subsets of modalities X3 where the abbreviations of subsets are follows: M: MNIST; S: SVHN; T: Text; M,S: MNIST and SVHN; M,T: MNIST and Text; S,T: SVHN and Text; M.S,T: all. We report the means and standard deviations over 5 runs.
MODEL
M
T
M,S
M,T
S,T
M,S.T
MVAE 0.90=0.01
0.44++001
0.85=0.10
0.89
0.97=0.02
0.81 +0.09
0.96#0.02
MMVAE
0.95=0.01
0.79+.005
0.99_o.01
0.87
0.93=0.03
0.84=0.04
0.86440.3
MOPоE
0.001
0.80
0.99 0.01
0.97 1:0.1
0.98
0.99 1.001
0.98
Table 3: Generation coherence for MNIST-SVHN- For conditional generation, the letter above the horizontal line indicates the modality which is generated based on the subsets X5 below. We report the mean values over runs. Standard deviations are included in Appendix C.3
M
T
MODEL
JOINT
$
T
S,T
M
T
M,T
M
S
M,S
MVAE
0.12
0.24
0.20
0.32
0.43
0.30
0.75
0.28
0.17
0.29
MMVAE
0.28
0.75
0.99
0.87
0.31
0.30
0.30
0.96
0.76
0.84
MoPoE
0.31
0.74
0.99
0.94
0.36
0.34
0.37
0.96
0.76
0.93
4.2 POLYMNIST
The PolyMNIST dataset consists of sets of MNIST digits where each set { consists of 5 images with the same digit label but different backgrounds and different styles of hand writing. An example of one such tuple is shown in Figure 2 Thus, each "modality" represents a shuffled set of MNIST digits overlayed top of (random crops from) 5 different background images, which are modality-specific In total there are 60, 000 tuples of training examples and 10, 000 tuples of test examples and we make sure that no two MNIST digits were used in both the training and test set
2
3
9
3
4
5
6
7
8
9
2
3
4
5
6
7
\
4
5
6
8
5
Figure 2: Ten samples from the PolyMNIST dataset. Each column depicts one tuple that consists of five different "modalities".
The PolyMNIST dataset allows to investigate how well different methods perform given more than two modalities. Since individual images can be difficult to classify correctly (even for human observer) one would expect multimodal models to aggregate information across multiple modalities. Further, this dataset facilitates the comparison of different models, because it removes the need for modality-specific architectures and hyperparameters. As such, for a fair comparison, we use the same architectures and hyperparameter values across all methods. We expect MMVAE and our proposed method are able to aggregate the redundant digit information across different modalities, whereas the MVAE should not be able to benefit from increasing number of modalities, because it does not aggregate unimodal posteriors. Further, we hypothesize that the MVAE will achieve the best generative performance when all modalities are present, but that will struggle with an increasing number of missing modalities. The proposed MoPoE-Va VAE should perform well given any subset of modalities.
that both the
PolyMNIST results Figure compares the results across different methods. The performance terms of three different metrics shown as a function of the number of input modalities; for instance, the log-likelihood of all generated modalities given one input modality (averaged over all possible single input modalities). As expected, both the MVAE and MoPoE- VAE benefit from more input modalities, whereas the performance of the MVAE stays flat across all metrics. In the limit of all 5 input modalities, the log- likeliiiddood of MoPoE. VAE is on par with MVAE, but the proposed method is clearly superior in terms of both latent classification well as conditional coherence
Details how the dataset was generated are included in Appendix D

---

Published as a conference paper at ICLR 2021
Table Test log- si-liikilihods MNIST-SVHN-Tex. We report the test log-likelshoods of the joint generative model conditioned on the variational posterior of subsets of modalities is xx)) (# MNIST; zg: SVHN; 2T: Text; X (aM, æs Fr)).
MODEL
X
X
X
X
X M xs
X(eM; 2T
X 2s ar
MVAE
-1.90
-2090
-1855
-2133:669
-182542.
-2050=26
-0855
MMVAE
-1.41
-1987 1.5
1257
-2018
-9122473
-2002
-1955
MоPоE
-1819a5.7
-1991428
-1588 6.2
-2024-226
11222
-1987 #3.1
-1850
Cpppiicatoo Accuroy
30000
0525
0.5
9000
Mallol
Log Uiwloood
MMAL MMVVEE MURIE
-500
0.50
4030
Corditoral
MYAE MIVWAE
McPeE
-220
Number input medalities
Namber modalties
Number of input modalickss
Figure 3: Performance on PolyMNIST as a function of the number of input modalities, averaged over all subsets of the respective size, Performance is measured in terms of three different metrics (larger is better) and markers denote the means (error bands denote standard deviations) over five runs. Left: Linear classification accuracy of digits given the latent representation computed from the respective subset. Center: Coherence of conditionally generated samples (excluding the input modality). Right: Log-likelihood of all generated modalities. Not shown: The joint coherence is 3.6 (+1..) 20.0 (=119), and 12.1 (=1) percent for MVAE, MMVAE, and MoPoE respectively.
across any subset of modalities. Analogously, in the limit of a single input modality, MoPoE- VAE matches the performance of MMVAE. Only in terms of the joint coherence (see figure legend) the MMVAE performs better, suggesting that a more flexible prior might be needed for the MoPoE- VAE. Therefore, the PolyMNIST experiment illustrates that the proposed method does only theoretically encompass the other two methods, but that it is superior for most subsets of modalities and even matches the performance in special cases that favor previous methods.
4.3 BIMODAL CELEBA
In this dataset, the images displaying faces (Liu et al. 2013) are equipped with additional text de- scribing the faces using the labeled attributes. Any negatively labeled attribute is completely missing in the string which makes the text modality more challenging. Compared to previous experiments, additionally use sopelity- latent spaces, which were found to improve the generative quality of model (Hsu & Glass 2018 Sutter et al. 2020; Daunhawer et al. 2020 P Figure dis- plays qualitative results of images which are generated given text. Table 5 shows the classification
results for the coherence of generated samples as well as the classification of latent representations. We see that the proposed model is able match the baselines on this challenging dataset, which favors the baselines, because it consists of two modalities. [ggure/4shows that attributes like "gen- der" or "smiling" are learned well, as they manifest generated samples and can identified from the latent representation. Subtle and rare attributes more difficult to generate consistently; evaluations specific to the different labels are provided in Appendix E.3
5 CONCLUSION
In this work, propose new multimodal ELBO formulation. Our contribution is threefold: First, the proposed MoPoE- VAE generalizes prior works (MVAE, MMVAE) and combines their benefits. Second, we analyze the strengths and weaknesses of previous works and relate them directly their
'For details, see Appendix
8

---

Published as a conference paper at ICLR 2021
Table 5: Classification and coherence results on the bimodal CelebA experiment. For latent rep- resentations and conditionally generated samples, we report the mean average precision over all attributes (I: Image; T: Text; Joint: and T).
LATENT REPRESENTATION
GENERATION
MODEL
1
T
JOINT
I
T-I
MVAE
0.30
0.31
0.32
0.26
0.33
MMVAE
0.35
0.38
0.35
0.14
0.41
MOPOE
0.40
0.39
0.39
0.15
0.43
Bignow bol rede. care no bea1d wes
Atractive
Sings non Hgp
Socled Vredow bup
Meathilggth open rm weein eanting. Forict yourg
sigh ireati ipnly
cheisooess night/yppis tox. salfit
beep hiedd iralois beadd
Iesheey boad
benpp brow ralio no
brow hol- beard wau
rals, nouh
no tali
weetig
lith open
butyy
westing
faxe sniling.
crube
hat serring
resssfe
Figure Qualitative results for bimodal CelebA. The images are conditionally generated by MoPoE-VAL using the text on top of each column.
objective and choice of posterior approximation function. Finally, in extensive experiments we em- pirically show the advantages compared to state-of-the-an models and even match their performance on tasks that favor previous work. In future work, we would like to evaluate previous extensions to multimodal VAEs. Addtionally, will explore different types and combinations of abstract mean functions and investigate their effects on the model and its performance as well as their theoretical properties (e.g., tightness) compared to existing methods.
ACKNOWLEDGMENTS
We would like to thank Ričards Marcinkevits for helpful discussions and proposing the name "PolyMNIST". ID is supported by the SNSF grant #200021
REFERENCES
Tadas Baltrušaitis,, Chaitanya Ahuja, and Louis Philipp Morency. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41 (2):423 443, 2018
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. In Thirty-Seoond AAAI Con- ference on Artificial Intelligence, 2018.
Yuri Burda, Roger B Grosse, and Ruslan Salakhutdinov. Importance Weighted Autoencoders. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http: http///aiiii...///.///ee//...t./ org/ abs/1509. 00519
Thomas M Cover and Joy A Thomas. Elements of Information Theory (Wiley Series in Telecommu- nications and Signal Processing). Wiley-Intersciene USA, 2006. ISBN 0471241954.

---

Published as a conference paper at ICLR 2021
Imant Daunhawer, Thomas M Sutter, Ricards Marcinkevics, and Julia E Vogt. Self-supervised Dis- entanglement of Modality-speciffe and Shared Factors Improves Multimodal Generative Models. In German Conference Pattern Recognition. Springer, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. ICLR, 2(5):6, 2017.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(():1771-1800 2002.
Wei-Ning Hsu and James Glass. Disentangling by Partitioning: Representation Learning Frame- work for Multimodal Sensory Data. 2018.
abs 11264
Xun Huang, Ming-Yi Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 172-189, 2018.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descrip- tions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3128-3137,201
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 141211412066 2014.
Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. CRLhhttp:/aarx: wrt. absg//bs//nn /1tp///11.... 6114
Richard Kurle, Stephan Günnemann, and Patrick van der Smagt. Multi-Source Neural Variational Inference. In The Thirty-Thire Conference on Artificial Intelligence, AAAI 2019, pp. 4114-4121. [AAAT} Press, 2019. doi:
URL https:://oi..ogg///
33014114
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. http::t/:///an.
lecun. www. excb/mnistt
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep Learning Face Attributes in the Wild.
In The IEEE International Conference on Computer Vision (ICCV), 2015.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-IO), pp. 807-814, 2010.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
C Niculescu and L Persson. Convex Functions and Their Applications: A Contemporary Approach. 2005.
Frank Nielsen. On the Jensen-Shannon symmetrization of distances relying abstract means. Entropy, 2019, ISSN 10994300. doi: 10.3390/e21050485..
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, and others. Scikit- learn: Machine learning in Python. Journal of machine learning research, 12(0ct):2825-2880, 2011.
Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philipps Morency, and Barnabás Póczos. Found in translation: Learning robust joint representations by cyclic translations between modalities. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 6892-6899, 2019.
10

---

Published as a conference paper at ICLR 2021
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31th International Con- ference Machine Learning, ICML 2014, Beijing. China, 21-26 June 2014, pp. 1278-1286, 2014.
abs 0albb///411400000
Alice Schoenauer-Sebag, Louise Heinrich, Marc Schoenauer, Michele Sebag, Lani F Wu, and Steve J Altschuler. Multi-domanl adversarial learning. 7th International Conference on Learn- ing Representations. ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 New Orleans, LA, USA, May 6-9, 2019, 2019. URL
http://awwwrrrgaaaaaa/oo////o/ot / abs / 903 09239
Yuge Shi, Siddharth, Brooks Paige, and Philip Torr. Variational Mixture-of- Experrs Autoencoders for Multi-Modal Deep Generative Models. In Advances in Neural Information Processing Sys- tems, pp. 15692-15703, 2019.
Yuge Shi, Brooks Paige, Philip H S Torr, and N Siddharth. Relating by Contrasting: A Data- efficient Framework for Multimodal Generative Models. 2020. Jtt/sttss://arxiv.co
abs/2007 01179
Thomas M Sutter, Imant Daunhawer, and Julia E Vogt. Multimodal Generative Learning Utilizing
Honsen-Shannon-Divergnnn 2020. URL
08242
Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Joint Multimodal Learning with Deep Generative Models. pp. 1-12, 2016. Uttpttp: org/ wwwpabs/////ooooooorot...oee.aer absaabas//1111 /111111111111 01891
Yingtao Tian and Jesse Engel. Latent translation: Crossing modalities by bridging generative mod- cls. 2019. URL https: /ttp:p:://aaxi..oo wrw..a/////.........././ 08261
Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis Chiii-Phillppe Morency, and Ruslan Salakhut- dinov. Learning Factorized Multimodal Representations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
www.eeerreeiee..ee//orrraraa woreet/ooommiiiommm d wow.rr-rgggggyyoo
Ramakrishna Vedantam, Ian Fischer, Jonathan Huang, and Kevin Murphy. Generative Models of Visually Grounded Imagination. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL htttps:///ppeeeeeeeeee
Mike Wu and Noah Goodman. Multimodal Generative Models for Scalable Weakly- Supervised Learning. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, 3-8 December 2018, Montreal, Canada, pp. 5580-5590, 2 2018. URL /ttttp:://xxx...../ org/ abaaaaas///// 05335
Mike Wu and Noah Goodman. Multimodal Generative Models for Compositional Representation Learning, 2019.

---

Published as a conference paper at ICLR 2021
A PROOFS
A.1 PROOF OF LEMMA
Ex DKL ( (=))()))), i.e., the sum of KL-divergences in Equation 5 can be used for describing the joint probability Po))..X..
Proof. We show that the convex combination of KL-divergences can be directly related to the joint probability log (X)):
Σ
DKL Dx = Σ
(x(x))
log
Pa(2|XX
X E((
log ( (X+x)
Σ
+ log pa (X))
(11)
ps (z, X)
which can be reformulated as an expression of the joint probability log log (X):
pa (x((x))po(2)
Po (X)
DKL (To (po +
E (=(((
log
2M
2M
T (5x)
XxEP(X)
X EX()
Equation
From the non-negativity of the KL-divergence, we derive the lower bound to the joint probability log p(X):
1
log (X) 2 2M
(x log @0(2 xxx
(12)
A.2 PROOF OF LEMMA 2
LMoPOE d; X) multimodal ELBO, that is (x( LooraE d; X).
Proof. The sums using index & also over all M subsets in the power set P(X). We use k only for better readability.
log p() = DKL ( + (x flog p(=,X)
(13)
(z(x
p(z, X)
DRL.
Σ 7(2 ((=1xx))lx<<(x0)
+ E zKr En, [(=138))
log
(14)
2M
Σ
((x Xx)
p(2,X) X)
2 E zur
Ex [(
log
Σx 7(2 (xx)
(15)
2
(L/o (x|2)] (1x)
DKIl 2k
Σ
To. (x(Xx))lpo(x))
(16)
XxEP(x)
C(0, X)
(17)
A.3 PROOF OF LEMMA
Maximizing LMOPLE (9, 6: X minimizes the convex combination of KL-divergences of the powerset
P(X) given in Equation
12

---

Published as a conference paper at ICLR 2021
Proof. Lemmal shows that the sum of KL-divergences in Equation (5) is able to describe the joint probability and can be used to form valid ELBO. Equation (12) is the convex combination of ELBOs given subset's posterior approximation (=((2(X)) Utilizing Jensen's 's inequality, it follows:
(x(x Lo (x
log pa (x(
Σ XLEP(X)
log
S IE
(18)
M
2 Ln (0(x)x))
Σ 96 (z)
2
where the sums the right hand also iterate over all X P(X). From Equation the proposed LMoPOE(B Φ; X) is not only a valid lower bound to the joint log-probability log (X), but also a tighter one than the convex combination of ELBOs:
18
we that
log Po Px(X
log pe (X) E 2 Ex (2(()e)
(19)
2 En 90 (xb))
LMoPot (0, Φ; x)
(20)
pa 2 ppo
(x(xx) (xEP(8)
log
(21)
2M
(z
As Equation (99 can be directly derived from Equation maximizing the proposed objective results in minimizing the convex combination of KL-divergences.
In the following, we derive the inequality in Equation
18 more detail:
Σ
Eg (xI(a
log
Po (X (z)
(22)
2M
T xx)
X (X)
Basls(Xe) Pl (x|=)po - log (z|xx))
(23)
2M
(xEP(X)
Σ
(x((x) log (o
(x(() olog (x(xa)))
(24)
2M
2M
EX()
X.EP(X)
=B
zdr
>i
2B ghr Ex (2(4)1))
[los| giv Ee do(x
]
E 2
Σ. do (x xx) (X
2tr E. (x(())
log
2
Σ (d((|({))
(25)
E 2M Σx (0 xx)
log (p=(X(2))0(2)) log
2M Σ (x((
(26)
pe (x 22 k)pp
E 20 Ee [x)
log
(27)
2 k g0 (x
In the minuend of Equation (24). the ordering of expectation and sum can be exchanged due the linearity of the expectation. In the subtrahend of Equation (24). the sum of expectation of the pos- terior approximations of subsets can be reformulated into the expectation of a mixture distribution using Jensen's 's inequality. Due to the convexity of the function f(t) t log (Cover & Thomas 2006] p.29), the expectation of mixture distribution is lower bound to the sum over the expec- tation of posterior approximations as the mixture distribution can be seen as convex combination of posterior approximations of subset of modalities T (=(Xx). Hence, the inequality from Equa-
tion 24 to Equation (25) follows as we decrease the subtrahend in Equation 25)
13

---

Published as a conference paper at ICLR 2021
B EVALUATION OF EXPERIMENTS
For the experiments, we evaluate all models regarding three different metrics: the classification accuracy (or average precision for CelebA) on the latent representation, the coherence of generated samples and the test set log- rog-likelihoods.
The latent representations are evaluated using a logistic regression classifier from scikit-learn (Pe- dregosa et al. 2011). The classifier is trained using 500 samples from the training set which are encoded using the trained models. The evaluation is done on the full test set and the reported num- bers are the average performances over all batches in the test set.
The generation coherence evaluated using the same networks as the unimodal encoders which trained beforehand. For every data type, train a neural network classifier in a supervised way. The architecture the classifier identical to the encoder except from the last layer. For joint coherence, all generated samples having the same label, they are considered coherent. The coherence accuracy is the ratio of coherent samples divided by the number of generated samples. For conditional generation, the conditionally generated samples have to be coherent to the input samples.
evaluated by the classifier and if all modalities
classified as
The test set log-likelihooss are evaluated using 15 importance samples for all models and the ported numbers are the averages over all test set batches.
stated differently, the reported numbers in section are the mean and standard deviations of runs with different random seeds. All models evaluated use the same architectures and numbers of parameters. The likelihoods of the different modalities are weighted to each other according to the size of the modality for all experiments. The most dominant modality is set to 1.0. The remaining ones are scaled up by the ratio of their data dimensions. For example in the MNIST-SVHN-Tew experiment, SVHN is set to 1.0 and MNIST to 3.92 which is the ratio of their data dimensions.
For all unimodal posterior approximations, we assume Gaussian distributions N2; μt, の² I²I,) where is the number of latent space dimensions. In all experiments, the mixture components are equally weighted with #components
B.1 COMPARISON TO PREVIOUS WORKS
Shi et al. (2019) in the end use a different ELBO objective including importance samples LIWAE (Burda al.) (0166). We compare all models without the use of importance samples these could be easily introduced to all objectives and are not directly related to the focus of this work which is choice of joint posterior approximation.
Sutter et al. (020) utlize the Sensen-Shanmer divergence a regularizer instead of the KL- divergence. This results the use of a dynamic prior and shows promising results. Besides the dynamic prior, they model the joint posterior approximation as well using a MoE. Again, we do not include models utilizing dynamic prior as this could be introduced to all formulations and is not the focus of this work.
B.1.1 EQUIVALENCE TO ELBO FORMULATION IN SHI ET AL. 2019
For clarity, we show here the equivalence of the formulation in Equation (0) to the formulation in (Shi et al. 2019 p.5).
14

---

Published as a conference paper at ICLR 2021
(0,E(0,0;3) (X]] DKL
1 M
Σ
gos (2| lipo (z)
(28)
log M E 1= 10y (z)a0))
[k E 1 د=== nw (z)(n))
(29)
Pe (2)
log (x(x(x))
[x (=1x))
(30)
Pa (z)
Pa (z)
(x1x) log (x]=)] E (x(x))
log
(31)
(0(=(x))
log po (x(=) (z)
(x(()
(32)
(x(2|XX) X
M M
log Pe (x x (z) [2 X)
Eas (x/æ,)
(33)
where Equation The different formulation on the second line the first equation on page is coming from their use of importance samples.
32 and Equation
33
equivalent to the first equation page 5 in Shi et al.
15

---

Published as a conference paper at ICLR 2021
Table 6: Generation Coherence for MNIST-SVHN-TeX. For every subtable, the modality above the wide horizontal line is generated based on the subsets below the same line excepxcept for joint coherence. The abbreviations of the different modalities are follows: M:MNIST; S: SVHN; T: Text. Combinations thereof separated by commas result in the subsets consisting of the modalities. We report the mean value and standard deviation 5 runs.
MVAE
0.2410.01
0.20
0.32:10.03 0.87400.5
MMVAB
0.75+-0.0
0.99-40.01
MoPoE
0.74
0.99::0.01
0. 0.94
MODEL
M
M,T
MVAE
0.43+0.02
0.30++0.05
0.75
MMVAR
0.31
0.30
0.30.20003
MoPoE
0.3620007
0.34:10.06
0.37:10.6
MODEL
M.S
MVAE
0.28110.06
0.17:10.02
0.29 06
MMVAE
0.76:10.04
0.84
MoPoE
0.96
0.76#+0.0
0.93
MODEL
JOINT COHERENCE
MVAE
0.12:10.02
MMVAE
0.28++0000
C MNIST-SVHN-TEXT
C.I DATASET
The dataset MNIST-SVHN-Tex. was introduced and described by Sutter et al. (2020). Equal to Shi al. (2019) their bimodal experiment, we create 20 triples per set resulting in many-to-many mapping.
C.2 EXPERIMENTAL SETUP
The latent space dimension is set to for all modalities, models and runs. The results in tables 2 generated with 3 5.0. We train all models for 150 epochs. We use the same architectures in Sutter et al. (2020). For MNIST encoder and decoder, we use fully-connected layers, for SVHN and text encoders and decoder feed-forward convolutional layers. For all layers, use U-actvvctvo functions (Nair & Hinton, 2010). The detailed architectures can also be looked up in the released code. We Adam optimizer (Kingma & Ba 2014) with an initial learning rate 0.001.
C.3 ADDITIONAL RESULTS
In table we show the coherence results including the standard deviation of the runs which were removed from main part due to space restrictions.
Additionally, we perform the analysis of coherence in relation to log-likelihood for conditional gen- eration well, similar to the example using random generation in section 4.1 The combination of coherence and log- liglliiels shows the ability of model to learn the data distribution as well as the generation of coherent samples. Every point refers to different 3 value. We evaluated the mod- els for 3 [0.5, 1.0, 2.5, 5.0. 10.0, 20.0). The points in the figures the mean values of 5 different runs with the lines being the standard deviations both directions, coherence and log-likelihoodk
Figure 7 displays qualitative comparison between the three models using 100 randomly generated samples. The generated samples correspond to the numbers in section 4. MVAE is able to best approximate the joint distribution in terms of sample quality for the price of limited coherence,
16

---

Published as a conference paper at ICLR 2021
Coherancc Accuracy 12
CulAAOon AuQUrAc
MORE
MABE
MODE
Lnglitalhood
Log Liaed hold
Log Lighieetooo
M.S +T
(b) M.T S
(,) -
The three figures show the evaluation for the conditional generation of single modality given the other two in relation to the joint ligelikeldhood given these two modalities, e.g. in the first row we generate SVHN samples conditioned on MNIST and Text. The points in the figures are the mean values of 5 different runs with the lines being the standard deviations in bopth directions, coherence and ligeiiielssoods.
Figure 5: Coherence and Log-Likelihoods for MNIST-SVHN-Tex.
Mise
cole uno A400
-120
-300
-3000
Log reedd
Log Lheeeeo
(a) M + S
(b) +T
Mase
Hercos
Lng-Jcelhhod
soolloodd
(c) + M
(d) S + T
10
M056 MEVAZ MoY0E
rervaz
Hisot
Coho SECC ROUUSCI
Liveltood
Log Lilelioodd
(e) T + M
(0) T +
Figure 6: Coherence and Log- Likeliielssoods for MNIST-SVHN-TeN The three rows of figures show the evaluation for the conditional generation of two modalities given the remaining one in relation to the joint log-likelihood given this single modality, e.g. in the first row we generate SVHN and Text samples conditioned MNIST. The points in the figures are the mean values of different runs with the lines being the standard deviations both directions, coherence and log-likelihoods.
while MMVAE shows higher coherence but limited sample quality. MoPoE approximate MVAE's sample quality with start-of-the-art coherence.
17

---

Published as a conference paper at ICLR 2021
7 3
4
6
7
9
9
9
0
9
(a) MVAE: MNIST
(b) MVAE: SVHN
MVAE: Text
7
9
O
4
9
0
4
2
F
9
5
(d) MMVAF: MNIST (c) MMVAE: SVHN (f) MMVAF: Text
7
3
2
6
(g) MoPoE: MNIST
(h) MoPoE: SVHN
(i) MoPoE: Text
Figure 7: Qualitative comparison of randomly generate MNIST-SVHN-TE samples.
In addition the theoretical proof of Lemma 3 that Definition T miiiimizes the convex combination of ELBOs, we compare the performance of a model trained using the objective in Equation (5) to the proposed method MoPoE-VAE VAE. It can be seen that MoPoE-VAs VAE achieves competitive results to the model which is optimizing Equation (5). This shows empirically that the proposed method is indeed minimizing the convex combination of ELBOs in Equation 5 Equation (5) is extensively
minimizing the ELBO of every possible subset. Hence, Equation expensive to optimize.
is computationally much more
Table 7: Comparison of objectives: Equation (5) and Definition 1 We report the test set log- likelihoods of the joint generative model conditioned on the variational posterior of subsets of modalities (2 (æ MNIST; æz: SVHN; xr. Text; X (æ æs, (T))). For both ob-
jectives we use 2.5
MODEL
X
X|æs
X
X SL Øs
X|æM, ØT
X X| ØT
EQ. 5
-1810
-1993
-1831
-2039
-1811
-2000
-1839
MoPOE
-1815
-1990144
-1858
-2024=1.2
-1819
-1986
1848
18

---

Published as a conference paper at ICLR 2021
D POLYMNIST
D.I DATASET
For the creation of the PolyMNIST dataset, we fuse each MNIST image with random crop of size 28x28 from the background image of the respective modality. In particular, we binarize the MNIST image and invert the colors of the random crop at those locations where the binarized MNIST digit is visible. We use the following background images:
1. John Burkardt. Licensed under GNU LGPL. https://peoples sc. fsu fsu.edu/
http:///gggraccaatt_eeeeee jpg retreeed
retrieved 27.09.2020)
2. Edvard Munch. The Scream. Public domain. http::////woooo..wwweeeeeeee wik.meii......mm.mmmmmmm org/
Scream jpo [Online; retrieved
retrieeeetttreeee 27 27.09.2020]
3. The Waterloo Image Repository. Lena.
Copyright belongs to the author. [Online;
http://links uwaterloo ca trieved 27.09.2020]
4. John Burkardt. Licensed under GNU LGPL.
https://people.rt sc. fsu. edu/
- jjuurkartt hatts/arrdddtt///aaaaaaaaa data/ http://aaaaa////taaaaa field. jpg retiered
5. John Burkardt. Licensed under GNU LGPL. https::/people.scc fsu edu/
/
jpg
[Online; retrieved 270002200)
D.2 EXPERIMENTAL SETUP
The latent space dimension is set to 512 for all modalities, models and runs. All results in are based on B 2.5, which was found to be a reasonable setting for all models. for all methods and train all models for 300 epochs. We use an Adam optimizer (Kingma & Ba 2014) with an initial learning rate 0.001. The architecture is based on straightforward convolutional neural networks (without bells and whistles); for details, refer the released code.
use the same architectures
D.3 QUALITATIVE RESCITS
In Figures 8 to
II
we show qualitative results comparing the different methods.
0
/
2
7 5
/
3
4
5
8
0
2
3
2
9
7
7
/
4
0
/
8
0
/ /
2
3
9
6
7 7
&
9
3
C
S
7
0
0
3
8
7
9
3
S
7
0
2
4
5
6
7
9
2
3
4
6
6
4
9
0
7
0
R
5
8
9
2 O
3
4
8
6
7
2
3
/
3
34
6
8
4
0
/
3
8
9
L
3
4
5
7
4
0
/
3
5 6
7
8
21
0
7
0
(a) MoPoF
(b) MMVAE
(c) MVAE
Figure 8: Reconstructions across all modalities for all models. In every pair of rows, show one row of test images followed by one row of respective reconstructions.
19

---

Published as a conference paper at ICLR 2021
3
9
7
?
0
7
3
0 9
3
9
6
0
7
C
0
7
7
y
(a) MoPoE
(b) MMVAE
(c) MVAE
Figure 9: Ten unconditionally generated images from the respective five modalities for each model. Column-wise, we use the same latent codes, sampled from the prior. Note that, row-wise, the digits should not be ordered.
3
6
7
&
9
0
0
O
7
7
0
3
7
9
0
7
2
0
9
6
7
9
0
9
0
5
0
6
7
(a) MoPoE
(b) MMVAE
(c) MVAE
Figure 10: Conditionally generated images of the first modality given the respective test example from the second modality shown in the first row. Column-wise, we take different samples from the approximate posterior, which should result in stylistic variations for generated outputs, but which should ideally not change the digit labels.
9
Q
8
5
0
7
2
0
2
3
4
5
6
7
8
9
O
3
8
9
2
6
0
/
2
3
9
6
7
&
9
2
2
3
5
6
9
0
3
0
3
4
3
6
7 7
8
O
3
6
45
Q
3
4
5
6
8
O
7
3
5
2
5
6
0
3
4
7
8
O
7
3
7
5
0
3
5
7
8
O
3
0
5
0
3
9
3
7
8
O
3
8
3
5
3
8
A 9
3
4
5
7
8
O
7
6
7
0
3
4
3
7
8
0
3
6
3
8
5
6
0
3
3
6
7
8
0
3
6
4
2
6
0
3
7
8
0
2
5
2
3
7
(a) MoPoE
(b) MMVAE
(c) MVAE
Figure 1I: Conditionally generated images of the first modality given the four test examples from the remaining modalities shown in the first four rows. Column-wise, the approximate posterior, which should result in stylistic variations for generated outputs, but which should ideally not change the digit labels. Compared to the results from Figure [0] the MoPoE-VAI generates more coherent samples when conditioned on four instead of one input modality.
take different samples from
20

---

the high quality Published as a conference paper at ICLR 2021
E BIMODAL CELEBA
E.I DATASET
The bimodal version of CelebA was introduced by /Sttter al. (022)) The modality consists of strings which concatenate the attributes which are present in face. If an attribute is not present, it is not present in the string which makes it more difficult modality. Example strings can be seen in the top of Figure 4
E.1.1 MODALITY SPECIIIC LATENT SPACES
-pdalit spaces empirically have empirically shown to be useful (Bouchacourt et al. 2018 Hsu & Glass, 2018 Daunhawer al.) 2020 Sutter al., - for the generative qual- ity of samples. As CelebA is a visually challenging dataset, we adopt this idea and the ELBO for- mulation changes accordingly. For details, we refer the reader the beforehand mentioned papers. The latent space is divided into shared space 9s. (c|X) and modality-speciff spaces go,, (s, (m,)) for every modality æz. This allows every modality to encode information-wiln which is specific to this modality-in a separate latent space.
especially
M E (clx)) [Bas,, (1ylmn)) ploglllpppppe..... c)]l
E(0, o;
Σ
(34)
j=1
M DKL (90l, [2,)))))ppp)) DKL 2M
Σ To. (c|Xx)) (po(c))
where 2e. (c 2 Ex,ex ex do. (c( models the shared information and Io,, (sylarg) the modality-speciiff information for every modality.
All posterior approximations, shared and modality- are again assumed to be Gaussian dis- tributed, see Appendix B
E.2 EXPERIMENTAL. SETUP
The latent spaces set to 32 dimensions for the shared space well the podality-speccife spaces, resulting in 64 dimensions per modality in total. We set 3 = 2.5 for all runs and models. All models trained for 200 epochs. Again encoders and decoders both image and text use residual blocks (He et al. 2016). We use an Adam optimizer (Kingma & Ba 2014) with initial learning rate 0.0005. The architectures can also be looked up in the released code. The classification of samples and representations are evaluated using average precision due the imbalanced nature of the distribution of labels.
the same architectures in Sutter et al. (2020): the
E.3 ADDITIONAL RESULTS
We show the attribute-specific evaluations in figs. 12 and 13 where the representations and generated samples are evaluated specific to individual attributes. The evaluations are performed for all subsets of modalities. We see the differences in averagea precision between attributes in the coherence of samples as well as the latent representations. The correlation between learned representation and coherence of samples gives further evidence the importance of a good representation also for the multimodal setting and its task of conditional generation.
Figure 14 displays qualitative results of randomly generated samples. We samples the proposed model is able to generate which cover a wide variety of attributes. In the images, minor artefacts can be seen. This suggests that there is still room for improvement doing more rigorous hyper-parameter search.
21

---

Published as a conference paper at ICLR 2021
14
08-
0.6
Bo0 Undea 5y
Burry Syttroe
chilty
Octlee
Heph Chwalossees
Wacth Sightr Ppe
Nunce 5yo
Ove mit M81
Open
Doudle
(a) Img
08
64
0.2
0.0
lestod pydrroo
an 5000
Bkock MNr
chudb
Dobbl Cht
Gralee
Gnty Jai
high Crealtones
Narth sig Oyen
Matetee
NaTro Eyo
Is Beamt
vile 3i0
Nalo Moo
ban Cheets
weldce
Westing TOU
Oowrr ro
(b) Text
0.-
044
0.2
actrol settol bydoroo
d8 3000
Brom yet
nastr, gytemas
chulrr
Dosble Chn
Syeguneess
Coutee
Gnty jelr
hug Coabore
with Saynt 9oyntyyye pyen
Maron 5yo
to Beam
Fuoe
vole Jkn
Waay Wol
nicearro tiveee
bosu Cheett
wwee ro Eamtrog
weatee weldice
(c) Img and Text
Figure 12: Coherence of generated bimodal CelebA samples. For every subplot, image and text generated conditionally by the the modality subset of modalities in the caption. We see that different attributes are not learned equally well.
22

---

Published as a conference paper at ICLR 2021
14
0.8
04-
acrd the
soaan lite
Vasttree
Sye
weced proddig.aaaee
saitre
Siedaate
wart pet
werting, 1079
westin
Figure 13: Learned Latent Representations for the bimodal CelebA dataset.
(a) MoPoF: Img
(b) MoPoE: Text
Figure 14: Qualitative Results of randomly generated CelebA samples.
23


Reviews:
[{"review_id": "tePQKcX2vya", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**: The paper combines ideas from two previous works (MVAE and MMVAE) to propose a new multimodal formulation of the ELBO for VAEs. The approximate posterior consists of a mixture of subsets, with each subset a product of approximate posteriors for each modality. This generalizes previous works, and the authors show that their objective yields a lower bound on the full data log-likelihood. They also compare their approach with MVAE and MMVAE on three multimodal datasets, showing benefits in classification accuracy, coherence, and log-likelihood. Although the proposed method is not superior in all cases, it generally achieves a reasonable trade-off across performance metrics.\n\n**Strong Points**: Overall, the paper is fairly clear in its descriptions. For instance, it is clear that the mixture-of-products-of-experts (MoPoE) approximate posterior is a generalization of those from MVAE and MMVAE. This generalization aspect is one of the main selling points of the paper, as it illuminates the implicit assumptions/trade-offs in previous works. By combining MVAE and MMVAE under one framework, this may provide insights to researchers in this area.\n\nThe experimental analysis is thorough and rigorous. The authors use two recently proposed multimodal benchmark datasets and propose their own dataset, PolyMNST. The benefits of MoPoE is quantified on three metrics: classification accuracy, coherence, and log-likelihood. These are reported using subsets of the data modalities. A variety of experimental results are also shown in the appendix along with error bars (over 5 random seeds).\n\nThe authors provide experiment details, primarily in the appendix, for reproducibility. Although some aspects are omitted (e.g. architectures, optimizers, etc.), many details are present. Likewise, the fact that the authors report performance intervals using multiple seeds is helpful for reproducibility.\n\nThe paper is perhaps not entirely novel, as it simply combines the PoE from MVAE with the MoE from MMVAE. Nevertheless, identifying these generalizations and analyzing the performance trade-offs is still somewhat novel. \n\n\n**Weak Points**: Although the mathematical formulation is clear, this paper would benefit from a diagram and/or an algorithm box, which would help to further show the distinctions between the proposed approach and MVAE and MMVAE.\n\nI found some aspects of the motivation to be unclear. For instance, the authors begin their presentation of MoPoE by stating that they want to minimize a sum of KL divergences, using the power set of data modalities for the approximate posterior. It seems that this is a result of wanting to consider the case where some data modalities are missing. However, in the formulation and experiments, if I am not mistaken, all of the modalities are present, at least for training.\n\nThe authors also claim to want to restrict their formulation to models that are “scalable,” i.e. models that do not use a separate encoder per modality subset. Yet, in the experiments, the authors only investigate up to five modalities, and this is only coming from an artificially created MNIST dataset. Although I am not familiar with this particular experimental area, this does not strike me as a regime where “non-scalable” models could not be applied. The datasets, generally, also seem far removed from real-world multimodal datasets, e.g. audio and video.\n\nMoPoE does seem to improve the performance metrics in certain regimes, however, these improvements are, in many cases, incremental. That is, MoPoE does not enable any new capabilities over MoE and PoE. This is somewhat limiting in terms of the paper’s impact.\n\n**Accept / Reject**: I would lean slightly toward acceptance. The authors do indeed provide a generalization of previous methods, along with a thorough experimental analysis across multiple datasets and metrics. While the method is not particularly novel, it is well connected to previous works and well-situated in the literature.\n\n**Questions**: The formulation appears to rely on the assumption of missing modalities, however, if I understand correctly, all modalities are present during training. Would one expect larger differences in performance if modalities were missing during training? Is this a standard assumption in multimodal learning?\n\nIn Section 3.5, it is stated that MoPoE combines “the strengths of both MoE as well as PoE while circumventing their weaknesses.” Why is this the case? If this were the case, wouldn’t MoPoE perform uniformly better than MoE and PoE in all regimes?\n\n**Additional Feedback**:\n\nIntroduction:  \nUnclear what the purpose is in the opening paragraph. Representation learning in general?  \nIt would be helpful to define semantic coherence more precisely upfront.  \nSimilarly, abstract mean function should be defined more precisely upfront.  \n\nRelated Work:  \nAt what point does a model become non-scalable. Could non-scalable models handle the datasets in this paper?\n\nMethod:  \nTypically, the log marginal likelihood (top of page 3) is expressed as an average, not a sum.  \nShould explain what $q_\\phi$ is upfront. It is not mentioned until halfway through the next paragraph.  \nThe formulation involves all subsets of the power set, however, this seems to go against the assumption that some modalities are missing.  \nLemma 3: Should start with “Maximizing $\\mathcal{L}$…”\n\nExperiments & Results:  \nWhy should we care about coherence?  \nShouldn’t $\\beta=1$ yield the best log-likelihood estimate.  \nWhy is MoPoE not uniformly better (e.g. Table 2, 3, Figure 3, etc.)?  \nTable 4 caption should be “Classification and coherence results…”  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "eg3ejBgO-76", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes a new, scalable formulation for the ELBO for multimodal data which generalizes previous work. The key idea is to optimize over all possible subsets of modalities (their powerset), even if only a subset of the modalities are observed.\n\nQuality: The paper was good quality, as it was clearly written and easy to read. Experiments demonstrated the efficacy of the approach over existing baselines and the problem motivation as well as technical exposition was clear.\n\nSignificance: There is rather limited amount of work on generative modeling in multimodal applications, so this work addresses that gap.\n\nPros: This was a nice work which drew interesting connections between prior works (MVAE and MMVAE), provided explanations for each method’s strengths and weaknesses from a theoretical viewpoint, and proposed an approach which bridged the two. The writing was clear and easy to read.\n\nCons: It's not very clear to me how the PolyMNIST setting counts as having different modalities when the only thing that's changing is the background image. For the other datasets (e.g. MNIST-SVHN-Text), the separation is more clear.\n\n--------------\nUPDATE: Thanks for the clarification and the revisions to the PDF -- I will keep my score as is.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "EmY-4M4fvQN", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on providing a more generalize multimodal ELBO to encompass previous PoE and MoE as special cases and combines their benefits. To this end, the authors first define the new ELBO L_{MoPoE} which is an interesting extension of PoE and MoE. Different from PoE (product of experts) and MoE (mixture of experts), MoPoE (mixture of product of experts) explores a more general way by mixing more experts where each expert is the product of a subset of all modalities’ posterior. In this way, as illustrated by the author, PoE and MoE can be seen as specific cases of MoPoE easily. The proposed model achieves competitive results compared with PoE and MoE.\n\nAlthough the proposed MoPoE is exciting, the following concerns still need to be clarified to avoid misunderstandings:\n\nThe proof of Eq. 18. Jensen's inequality can be simply written as $f(E[x])≤E[f(x)]$, where $f$ is a convex function. The equation on the left hand can be seen as calculating the mean of $f(q(z|X))$ while on the right hand the mean of $q(z|X)$ is calculated first and then a function $f$ is used on it. According to my derivation, an error occurred in the direction of the inequality sign in Eq. 18. I hope that the author can check the inequality and give a more detailed derivation to avoid confusion. \n\nAt the same time, intuitively, I think Eq. 18 may be problematic. For the two implementations represented by Eq. 19 and Eq. 21, Eq. 19 is more concise and tighter. In my opinion, it is difficult for these two advantages to happen at the same time.\n\nMinor concerns:\n1.\tUsing all subsets of all modals is complicated. For example, when we have $n$ modals, $M$ is equal to $n^2$. It will be better for the authors to explain this in the paper.\n2.\tThe Sub-sampled Training Paradigm strategy used in MVAE (Wu & Goodman, 2018) has a similar effect to Formula 5. The author should discuss and analyze this.\n3.\tThe samples generated in Fig. 14 need to be discussed. \n4.\tRegardless of whether Lemma 3 can be proved, a verification test may be a good choice. An experiment implemented using Eq. 5 can be added to better illustrate the effect of the model proposed by the author.\n5.\tAlthough as the author stated that MoE is a special case of MoPoE, the loss function in Eq. 9 and MMVAE model (Shi et al., 2019) (the first equation on page 5) are different. The authors can give some explanations.\n\n----------------------------------------------------------------------\nUpdate after rebuttal\n----------------------------------------------------------------------\nThanks for the feedback.\nAlthough there are some minor issues, my main concerns are addressed. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "77ilZ-StlyU", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper formulates a multimodal ELBO as a mixture of product of experts. This allows them to use one encoder per mode while still allowing inference over any subset of modes without needing a new encoder for each subset. The idea is simple and appears to improve on baselines derived from a mixture or a product of experts.\n\nAt a high level I think the paper is fine, simple to understand, and the results are straightforward and good. However, the paper maybe dedicates too much space to explain this simple idea and not enough space analyzing the ELBO, e.g., looking at the tightness of the bound in comparison to baselines.\n\nSome remarks:\nP1:\n* First sentence talks about generalization, but it's not really substantiated in any of the experiments. It just seems thrown in there without a lot of support.\n* I have to start with that I disagree that VAEs are self-supervised models (or at least I don't gain anything by saying they are self-supervised generative models). I would omit this from the paper as it's confusing in the context of clearly self-supervised models (e.g., ADMDIM, MoCO, SimCLR, CPC, etc) and unnecessary. I'm happy to be convinced otherwise, but it might not be worth it for this paper.\n* So there's a lot of 2^M for the power set over modalities, but do you really use the null set? Maybe it should be 2^(M-1)?\n* Maybe explaining what abstract mean is because it's not a common term.\nP2:\n* Section 2: again, self-supervised\nP3:\n* 3.2: again, 2^M: is this correct?\nP4:\n* 3.3: \"sharper than any of its experts\" I feel like a citation is needed here. Mind going into a little detail why this is and why this is a problem?\nP5:\n* \"while circumventing their weaknesses\": could you clarify what you mean here?\n\nFinally, I'm curious if any analysis is possible on the lower bound, e.g., tightness compared to lower bounds used in MoE and PoE works.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After a bit of discussion, all reviewers are for accepting the paper.

Strengths:
+ Clarity (agreed by R4, R2, R1). The paper is easy to read and follow the core contributions. R3 had a concern about the correctness of a derivation, which was resolved in the discussion.
+ The work solves a core problem of generative models over multimodal applications, building on prior work with mixture and product of expert models. As R1 notes: "By combining MVAE and MMVAE under one framework, this may provide insights to researchers in this area."
+ On the datasets studied, the details for reproducibility are transparent, and multiple metrics and uncertainty over the metric results are reported.

Weaknesses:
+ Multi-modality of the benchmarks. The experiments evaluate on MNIST-SVHN, "PolyMNIST", and CelebA. The latter two benchmarks are fairly arguable in whether they're really multimodal as R4 notes: e.g., CelebA has two "modalities" of image and attribute pairs. It seems arguable whether you even need multimodal approaches.
+ Scale of the benchmarks. Language models (especially with Transformer architectures) have been studied quite a bit over multiple modalities, and these works scale significantly better applying simple strategies. It remains to be seen empirically what the utility of multimodal latent variable models really are.
Decision: Accept (Poster)