MMVAE+: Enhancing the Generative Quality of Multimodal VAEs without Compromises
Authors: Emanuele Palumbo
Imant Daunhawer
Julia E Vogt
Venue: ICLR 2023 poster (2023)
Keywords: Multimodal Variational Autoencoder
Variational Autoencoder
Multimodal Generative Learning

Abstract: Multimodal VAEs have recently gained attention as efficient models for weakly-supervised generative learning with multiple modalities. However, all existing variants of multimodal VAEs are affected by a non-trivial trade-off between generative quality and generative coherence. In particular mixture-based models achieve good coherence only at the expense of sample diversity and a resulting lack of generative quality. We present a novel variant of the mixture-of-experts multimodal variational autoencoder that improves its generative quality, while maintaining high semantic coherence. We model shared and modality-specific information in separate latent subspaces, proposing an objective that overcomes certain dependencies on hyperparameters that arise for existing approaches with the same latent space structure. Compared to these existing approaches, we show increased robustness with respect to changes in the design of the latent space, in terms of the capacity allocated to modality-specific subspaces. We show that our model achieves both good generative coherence and high generative quality in challenging experiments, including more complex multimodal datasets than those used in previous works.

Published as a conference paper at ICLR 2023
MMVAE+: ENHANCING THE GENERATIVE QUALITY OF MULTIMODAL VAES WITHOUT COMPROMISES
Emanuele Palumbo Imant Daunhawer? & Julia E. Vogt?
1 ETH AI Center, Zürich, Switzerland
2 Department of Computer Science, ETH Zürich, Switzerland emanuele pa umbo@ai. wtt.bo.aaieeh..........
ABSTRACT
Multimodal VABs have recently gained attention efficient models for weakly- supervised generative learning with multiple modalities. However, all existing variants of multimodal VAES are affected by non-trivial trade-off between gen- erative quality and generative coherence. In particular mixture-based models achieve good coherence only the expense of sample diversity and resulting lack of generative quality. We present a novel variant of the mixture-of -fixuure-ff-epperts multimodal variational autoencoder that improves its generative quality, while maintaining high semantic coherence. We model shared and modality-specifie information separate latent subspaces, proposing an objective that overcomes certain dependencies hyperparameters that arise for existing approaches with the same latent space structure. Compared to these existing approaches, we show increased robustness with respect changes in the design of the latent space, in terms of the capacity allocated to modality-speciff subspaces. We show that our model achieves both good generative coherence and high generative quality in challenging experiments, including more complex multimodal datasets than those used in previous works.
INTRODUCTION
Multimodal VAFs are promising class of models for weakly-supervised generative learning. Dif- ferent from initially proposed models in this class (Suzuki al. 2016 Vedantam et 2018). more
recent approaches (Wu & Goodman 2018 Shi et al.) 2019 2021
Sutter et al. [020]] 202T can
efficiently scale to a large number of modalities. These methodological advances enabled applica- tions in multi-omics data integration (Lee & van der Schaar 2021:
Minoura et al. 2021 and tumor
segmentation from multiple image modalities (Dorent et al.
2019).
Several variants of scalable multimodal VAEs have been proposed (Wu & Goodman 2018 Shi et al. 2019 |021| Sutter et al.) 202I) and their performance is measured in terms of generative quality and generative coherence. While generative quality measures how well model approximates the data distribution, generative coherence measures the semantic coherence of generated samples across modalities (e.g., see [hi al., 2019). High generative quality requires generated samples being similar to the test data, while high generative coherence requires generated samples to agree in their semantic content across modalities. For instance, in a dataset of image/caption pairs, conditional generation from the text modality should produce images where the depicted object matches the description in the given caption (e.g. matching color). Ideally, an effective multimodal generative model should fulfill both of these performance aspects. Still, recent work (Daunhawer et al.) 2022) shows that the predominant approaches exhibit non-trivial trade-off between the two criteria, which limits their utility for complex real-world applications.
In this work we focus on mixture-based multimodal VAEs, which show high generative coherence only at the expense of a lack of generative quality, a fact that undermines their performance in re- alistic settings. Results for models in this class are promising for capturing shared information, iee information that is communal across modalities on the underlying concept being described, while ex- hibit a lack of modelling of private variation, i.c. modality-spp modality-specific specific information for single modalities (see Shi al. 2019
Sutter al. 2021 Daunhawer al. 2022). In attempt enhance modelling

---

Published as a conference paper at ICLR 2023
of private information, recent work (Sutter et al. 2020) has suggested introducing poddality-specifff latent spaces in addition to a shared subspace for mixture- based multimodal VAEs. However, we revise such proposed extension and find that improvement in terms of generative quality comes again at the expense of reduced generative coherence. Most importantly we uncover a relevant short- coming of existing approaches with separate subspaces, in that we find generative coherence to be overly sensitive to hyperparameters controlling the capacity of private latent subspaces, which in practice calls for expensive model selection procedures achieve adequate performance.
Incorporating the idea of modelling the latent space as a combination of shared and modality-specific encodings. we propose the MMVAE+, variant of the mixture-of-exiiee exxertsss multimodal VAE (MM- VAE et al. (219g) with a novel ELBO, that significantly improves the diversity of the generated samples, without sacrificing semantic coherence. Compared to previously proposed models, our method achieves both convincing generative quality and generative coherence (Section 4.1) No- tably, its performance in terms of both criteria robust with respect to hyperparameters controlling latent dimensionality, compared to previous methods with separate shared and private latent sub- spaces. (Section 4.2) Finally, we show our proposed model can successfully tackle a challenging multimodal dataset of image and text pairs (Section 4.3) that was shown to be too complex for existing multimodal VAEs (Daunhawer et al. 2022).
RELATED WORK
Multimodal generative models are promising approaches for learning from co-occurring data sources without explicit supervision, by exploiting the pairing between modalities as form of weak-supervision. Previous work has achieved outstanding results for multimodal generative tasks such image-to-iman translation (Zhu et al., 2017: Choi et al. 2018) or text-to-image synthesis (Reed et al. 2016). While these models are designed for specialized tasks limited to a fixed number of modalities, in lot of real-world settings there is need for general methods that leverage large datasets of many heterogeneous modalities. A prominent example is the field of healthcare, where personalised medicine requires learning from large-scale multimodal datasets comprised of medical images, genomics tests, and clinical measurements. Multimodal VAEs (Suzuki et al. [2016] Wu & Goodman 2018 Shi al. 2019 2021, Sutter et al., 2020 202T are a promising model class for such applications, showing encouraging results towards efficient learning from multimodal datasets with multiple heterogeneous modalities.
Multimodal VAFs extend the popular variational autoencoder (VAE, Kingma & Welling, 2014) to multiple data modalities. Initially proposed approaches (Suzuki et al. Vedantam et al. 2018) lack scalability in the number of modalities, requiring an additional encoder network per possible subset of modalities, to enable inference from the given subset. Other proposed methods require explicit supervision (T'ai et al. 2019), which demands prior expensive data labelling processes. In contrast, recent methodological advances enable learning from a large number of modalities ef- ficiently and without explicit supervision, by using joint encoder that decomposes in terms of unimodal encoders. Previous work proposed three different formulations for the joint encoder; the product-of-experts (MVAE, Wu & Goodman 2018), mixture-of-exxprrs (MMVAE, Shi et al. 2019)) and mixture-of- oftpoooooooooooooooooooeeeeooooe
(MoPoE- VAE, Sutter et 2021).
Recent work (Daunhawer et al. 2022) shows that existing multimodal VAES exhibit tradeoff be- tween two desired performance criteria for multimodal generation, namely generative quality and generative coherence. While generative quality assesses the generative performance of the model for each modality, generative coherence (Shi et al. 2019) examines the learning of shared information by estimating the consistency in the semantic content between modalities in both conditional and unconditional generation. Daunhawer et al. (0022) show that existing product-based models exhibit low generative coherence, while mixture-basse based models exhibit a lack of sample diversity, which negatively affects generative quality (cp. Wolff et al. (021I.
Based on the three mentioned formulations of multimodal VAEs, subsequent work introduced ad- ditional regularization terms (Sutter et al. 22200 Hwang et al. 202I) and hierarchical latent spaces (Sutter& & Vogt, 2021 Vasco et al. 2022; Wolff et al. 2022). Previous work has also explored the possibility of assuming separate modality-sin codaaaly-sppccfic specific latent subspaces in addition to a shared subspace (Sutter et al. 2020; Lee & Pavlovic |021] Waan et al.) 2016). or leveraging mutual supervision (Joy)
let al. 2022) Yet, it is not clear whether these extensions overcome the fundamental tradeoff be-
2

---

Published as a conference paper at ICLR 2023
tween generative quality and coherence. For instance, for existing approaches to employ additional sppeeity-specific latent subspaces with mixture-based models, we observe an overly high sensi- tivity hyperparameters (Section 4.2), which raises fundamental questions with regard to model selection. Notably, we show that our proposed method-also in the class of multimodal VAEs with modality-specifft latent subspaces significantly more robust to changes in the same hyperpa- rameter values.
3 METHOD
3.1 PRELIMINARIES: MMVAE
Given set of M modalities X == B...... aM. multimodal VAEs learn latent-variable generative
model of the form pe ( 2) p(z) II
M -= po.,, (#mm (z), by using neural networks to approximate
the intractable evidence pe (X). Analogous to the VAE (Kingma & Welling 2014), multimodal
VAEs optimise lower bound of the log-evidence by maximizing the following objective:
pe Pe(X (X,z) x)
( yo x) log
go (=
where qs (z) ()) denotes the approximate posterior, a neural network that parameterized by T In the above form, the objective does not scale to large number of modalities (e.g., see Wu & Goodman 2018 Sutter et al., 202I) and additional assumptions are required for the joint encoder to be computed efficiently in terms of the unimodal encoders. Different formulations of the joint encoder have been proposed (Wu & Goodman 2018 Shi et al., 2019 Sutter et al.) (021). but here focus the decomposition used by the MM VAF (Shi al.] 2019), formulating the joint encoder Σm=1 Jo.. (z (m). When mixture
mixture-of-expprts of unimodal encoders go (=|x)=
distribution for the joint encoder is assumed, the expression in eqn. 11) is evaluated by computing the expectation with respect to each component the mixture distribution, and averaging over the results. As in this case we assume mixture-of-experts encoder, the MMVAE (Shi et al. 2019)
objective is sum indexed by the M unimodal encoders
1
M
pe (x( xx log qu (=
LMMAA 111M))
E myo (=(am)
(2)
M m=1
where
pe (X,z) log
M
Ego.. (2/am)
Eggn (=|mm)
log po... (0mlz)) +
pog (x,]1=) [z]]+ +
n=1 ngtm
p(z)
E qom (z m..) log
(x XX)
For each term in the sum, the encoding qom (z) ( is sampled from unimodal encoder, but used for the reconstruction of all modalities: the likelihood term encompasses the reconstruction well the reconstruc-
of the modality m, i.e. self-reconstruccion Ey (x) [log po,.. (Zmm|2))
tion across modalities, i.e. cross-modal reconstruction ΣM Em=1,u Byo... (=|=.) [log Po., (=r, Maximizing reconstruction likelihood of modalities not observed for inference encourages the la- tent code to contain information common across modalities, and can explain improved semantic coherence compared to previously proposed podduct-based approaches (see qualitative results in Shi jet al. (2019) and section 4.1.. However has also been recently been shown that the optimization of cross-modal reconstructions given unimodal encoding can lead to a lack of modelled diversity in private information for generation across modalities (Wolff et al., 2202T; Daunhawer et al. 2022) which manifests in average-looking samples with low generative quality (see section 4.1)
3.2 PRELIMINARIES: ADDITIONAL MODALITY-SPECITE SURSPACES
In an attempt to enhance modelling of private information and improve performance, recent work on multimodal VAEs (Sutter et al. (0200) has proposed to model the latent space for mixture-based
3

---

Published as a conference paper at ICLR 2023
models including modality-specifer latent subspaces. in addition shared subspace (Boucha- court et al. 2018 Tsai et al., 2019; Wang et al., 2016)- Formally, each modality æm is mod- elled to have its mppelity-specific latent code Wm. in addition to latent code shared between all modalities. The assumed generative model is then of the form pe(X, 2, W) p(z) [lm= I[=) P6.. (mm Wm (w)) where the shared latent code z and all the mppalit codes w1, W0a was =: W are assumed to independent. variational encoder qo (z, X is introduced to approximate posterior inference, and is modelled in line with the assumptions on the generative process, assuming conditional independence given X
go (z, xI q4_ (z X g (W)X) q4. (=|x) II 90wm (Wmm. (mm)
Again, achieving scalability in the number of modalities requires additional assumptions for the joint encoder q0, (= of the shared latent code In particular assuming mixture-of-expprrs, joint
encoder q== (= xx MV Em=1 a8em (z (m) one obtains a variant of the MMVAE with factorized latent space into shared and private subspaces
M
pe (e (X, . 2, W)
C CAMAAAE (AMMAE(TF1:M))
= M
Σ
E. Egox.. (=)(m))
log
qr_ (z w ( (X)
(4)
m=l
Now (WIX)
where
pe X W)
Eggen (=| gow (W)
log qu (=| X 19ow (WI))
E goam xm) Doreo mmm
[log pe.. (20 Iz, Wm)]+
M
E uoam (z cmm) Tounn (walma))
logge, (xn1 w.))
nym
E (=|or..)
log
qu (z) w (WIX)
90w (W) x)
Existing mixture-based multimodal VAEs with factor- ized latent space are based on this objective [Sutter et al. (2020). Note that both self-reconstrucction likelihood and cross-modal reconstruction likelihood terms are computed given the modelity-specific encoding from the targer modal- ity. This can seemingly be an advantage, given these addi- tional latent encodings can provide the modality-specific in- formation for reconstruction that is not modelled in the shared latent code. In particular, for cross-modal reconstruction no private information about the target modality can be present in the z encoding from the starting modality.
1.0
Generative colerence
0.8
0.6
0.24
12.3%
2306
5058
7036
1000
Modality-spociffo latent capacity
Figure 1: Shortcut problem for the objective in generative coher- ence as a function of of private la- tent capacity. The percentages the x-axis indicate the relative size of any private subspace with spect to the shared subspace.
However, we find that with such approach model perfor- mance is heavily dependent on hyperparameters controlling private latent dimensionality. In fact, when modality-speciff latent subspaces are given enough capacity, in terms of num- ber of dimensions, the modality-speciff encoding can contain all information for the given modality (i.e., both shared and modality-specifie features), while the shared subspace ig- nored by the decoders. As it was already observed in Waan al. (2016) in similar setting, we show here that this approach can lead to a degenerate solution subspaces and no shared sub- a consequence, to achieve sufficient gen-
a shorteut where all information flows through the modality space is learned for generation across modalities. erative coherence it is necessary to heavily constrain the size of private subspaces (see figure [T. Note however that this turn negatively impacts generative quality, being detrimental to modelling modality-specific variation, which was our motivation to employ additional private subspaces in the first place (see section 4.2) Therefore, assuming poddality-specfii latent subspaces seems to not be sufficient to overcome the trade-off between generative quality and generative coherence, and most importantly results in instabilities with respect to hyperparameters controlling private latent space capacity, which calls for expensive model selection procedures in practical settings.
4

---

Published as a conference paper at ICLR 2023
3.3 THE MMVAE+ OBJECTIVE
To enhance generative quality without compromising semantic coherence, we introduce the MM- VAE+ model, a novel variant of the MMVAE. Our model incorporates the idea of using factorized shared and modality-specific latent representations, and therefore makes the assumptions on the generative process and the variational approximation function presented in section 3.2 In order to ensure learning of a shared subspace while not incurring in a shortcut, auxiliary distributions for private features are used to facilitate the estimation of cross-modal reconstructions. The optimized objective is
M Σ
pom.. (2mm 22, Wm }p(z) (())))p(wn)) II m.-fmmm
LAMMAE (21:M) =
M
E
qoam (x (2mm)
log
q+0, (2) X) X)Mormn (W0m [m))
po., (2 Iz, t.a)
m=1
gown (1mll (38m) {wo rot (o (WF) Insem
(5)
where r1 (1(1)) ).. ..... T' (Was) are auxiliary prior distributions on moddllity-specinne features for each modality.
Comparing with the objective in the inferred modallty-sppeciff encoding is used only for self- reconstruction, and not for modal reconstruction, for which the sealaaity-secific code is sam- pled from the auxiliary prior. This crucial feature avoids the creation a shortcut in the presence of modality-specifff latent subspaces. In fact, cross-modal reconstructions are computed conditioning on a -on-informative value for the modality-specinn encoding, which forces the decoder to rely on the shared encoding to reconstruct unobserved modalities. As consquence, z encodes informa- tion useful to maximize reconstruction across modalities, i.c. shared information. Still, modality- specific subspaces can serve the purpose of modelling private variation, as private encoders are used for computing modality-specific codes for seff-reconstrucction.
The following lemma, for which we provide proof in Appendix proves that MMVAE+ optimizes a valid ELBO and therefore belongs to the family of multimodal VAES.
Lemma 1. The MMVAE+ objective (Equation 5 is valid lower bound on log pe (x1:)).
Note that while obtain valid ELBO for an arbitrary choice of auxiliary priors, we discuss our specific design choices in Appendix C As in (Shi et al. (2019), multi-sample version of the objective can provide tighter bound on the log-evidence. In Appendix D we discuss the details of the tighter version of the objective, along with the possibility to include a hyperparameter (Higgins
al. 2017) to weight latent space regularization.
EXPERIMENTS
We report experimental results for two challenging datasets introduced in previous work, namely PolyMNIST (Sutter et al. 202T) and Caltech Birds (CUB) Image-Captions (Shi et al. 2019 Wah et al., 2011). PolyMNIST is a synthetic dataset consisting of five modalities, depicting MNIST digits sharing the digit label, patched on random crops from five different background images, one for each modality. Each datapoint consists of five images, one per modality, where all the images share the digit label, but not the style the handwriting, as MNIST samples are shuffled prior the pairing. Shared information between modalities is therefore the digit label, while modality-speciff information includes the style the handwriting and the specific background image. The Caltech Birds (CUB) Image-Captions dataset consists of images of birds paired with matching linguistic descriptions. What makes this a challenging experiment is the large amount of modality-speciffc information for each modality, due the different nature of the information sources. While previous work (Shi et al. 2019 2021 Joy et al., 2022) has tackled simplified version of this experiment by using pretrained ResNee-features we train the models on actual images a more realistic setting that was shown to be challenging for the predominant approaches (Daunhawer et al. 2022). All quantitative results are averaged over three independent seeds and we report standard deviations. Technical details on the experiments and additional results are available in Appendices and G
Note that we refer to these as priors in the sense that they are not conditional on the data, but are effectively surrogate non ooninfforaaive inference distributions for private features. which do not depend on the observed data samples, and are different from the prior distributions for private information in the assumed generative process ((1r1),-.,p((u3))
5

---

Published as a conference paper at ICLR 2023
Oun
Wi
E (M....M))
(l.... M
2 (I....M)
f De 1.
N
(a) MMVAE
(b) MMVAF+ (ours)
Figure 2: Graphical models for comparison of MMVAE (left) and MMVAE+ (right). For both mod- els, gray circles represent observed variables, white circles represent latent variables, and non-cireled are the model parameters. Solid lines denote the generative process, while dashed arrows denote ap- proximated posterior inference with encoders. Note that for MMVAE+ cross cross-modal reconstruction does not depend on inferred mppelit information, thereby relying on the shared latent space to encode information useful across modalities, i.e. shared information. At the same time, max- imizing reconstrtttot of each modality m given the modalitt--peciinn Wm. code, encourages private variation to be encoded in mrodality-speciffe subspaces.
4.1 ENIIANCED GENERATIVE QUALITY FOR SEMANTICALLY COMIEENTT GENERATION
This section shows our results on the PolyMNIST dataset comparing MMVAE+ to the predominant approaches, namely MMVAE (Shi al., 2019), MVAE (Wu & Goodman 2018) and MoPoE-VAL (Sutter et al., (021). J. For completeness, we also compare with two recently proposed variations of MVAE and MMVAE, namely the MVTCAE (Hwang et al., 202T and the mmJSD (Sutter al. 2020), which both introduce additional regularization terms to the respective objectives. Qualitative results for conditional generation in Figure 3 illustrate the shortcomings of the existing approaches. The MVAE exhibits lack of generative coherence, whereas the MMVAE and MoPoE-VAR both show a lack of sample diversity, resulting poor generative quality. Even in this relatively simple setting, modality-specific information-such as the background details or the style of handwriting tends to collapse to expected values for cross-modal generation. contrast, the MMVAE+ shows significantly better generative quality with an improved sample diversity and high semantic herence. Comparing directly with the MMVAE, the results validate that our approach achieves enhanced generative quality without compromising semantic coherence.
For a quantitative comparison, we measure the performance in terms of generative coherence (Shi et al. 2019) and generative quality. The latter is estimated in terms of FID score (Heusel et al. 2017), a standard metric to evaluate sample quality for generative models in image domains. The scatter- plots in Figure 4 show that only the MMVAE+ reaches high generative quality and high semantic coherence consistently over a large range of hyperparameter values, while other models underper- form in one of the two performance criteria. Note that while MVTCAE and mmJSD both show some improvement over the predominant approaches they are based on, e.g. MVTCAE markedly improves generative coherence over the MVAF, they still exhibit poor performance in either condi- tional or unconditional genenration. Notably, the advantage of MMVAF+ persists for both uncon- ditional and conditional generation. Details for unconditional generation for our approach, which assumes separate latent subspaces, are at the end of Appendix
4.2 COMPARING WITH ALTERNATIVE APPROACHES WITH SEPARATE LATENT SUBSPACES
In this section, we compare MMVAE+ with alternative variants of multimodal VAE objectives that adopt a latent space that factorizes in separate shared and modality-specifie subspaces. In particu- lar, compare with the factorized MMVAE objective in well with its alternative version proposed in Sutter et al. (2020). which extends the single-space mmJSD model to a factorized latent space. In addition include DMVAE (Lee & Pavlovic 2021) I) in the comparison, which implements a product-basse objective with factorized latent representations. However, in contrast to the other compared models, its optimized objective is not provably an ELBO. Crucially, find that all these

---

Published as a conference paper at ICLR 2023
0
3
O
3
O
TO
(a) MVAE
(b) MVTCAE
(c) mmJSD
(d) MoPoE-VAL VAE (e) MMVAE
(I) MMVAE+
Figure? 3: Qualitative results for conditional generation on the PolyMNIST dataset. The input sample from the first modality is shown in the top row, and below are four conditionally generated samples for each of the remaining modalities.
1.0
1.0
Conditional Coherence
Uncoditional Coherence
0.8
0.8
0.6-
0.6-
0.2-
0.2
HENOO
0.0
0.0
250
200
150
100
250
200
150
100
50
Conditional FID
Unconditional
FID
Figure 4: Comparison of generative quality (in terms of FID, lower is better) and generative coher- ence (higher better). The left scatterplot evaluates conditional (cross-modal) generation and the right subplot quantifies unconditional generation performance. For each model, we show the results for range of representative I values (see Appendix F). An optimal model would lie in the top-right area of each scatterplot.
existing approaches exhibit a non-trivial problem: to improve generative coherence, they require strong constraining of modality-speciiff latent capacity, to avoid incurring in a shortcut where all in- subspaces. In figure
formation used by decoders is encoded independently modality we report an ablation for generative coherence for the compared models, varying the relative size of subspace for each modality with respect the shared subspace. Adopting any
the modality-speciinn of the existing approaches with separate latent subspaces, generative coherence quickly deteriorates capacity is allocated to private subspaces, reaching values close random when private and shared subspaces have equal size. This behaviour can arise from the fact that the encoding from subspace is used to compute both self- and cross-modal reconstructions the given modalities (see section 3.2). Therefore, with enough capacity, the modality-specific encoder can learn all information for the given modality (i.e., both shared and private features), which cre- ates a shortcut where all information flows through the modality-specine
each modality
subspaces and the shared subspace is ignored by the decoder. As additional evidence for this phenomenon, we report in Fig- ure 5b) latent classification accuracy (see Appendix [.2] for private subspaces, when varying private latent capacity. This metric shows the amount of shared information that a digit classifier can re- trieve from private embeddings. As expected, modality-speciff dimensionality is augmented, an increasing amount of shared information is present in the corresponding encodings. It is evident from the showcased results that current approaches achieve sufficient coherence across generated
Note that Wang al. 2016) propose dropout during training, instead of reducing private latent dimensionality, to avoid incurring shortcut. We address this in Appendix
7

---

Published as a conference paper at ICLR 2023
1.0
150
0.8
0.8
Generaaive qualty
Sund ensimi la péuuic
100+
0.5
0.6
FANOE
12.9%
2056
500%
757
10055
12.9%
2556
507%
759
100%
12,55
25%
71556
100%
Modality lasent capacity
Condality-specciee capucity
Modality specific latent cupacity
(a) Generative coherence (higher is better)
(b) Latent classification accuracy from modality-specifie encodings
(c) Generative quality (in terms of FID score, lower is better)
(lower is better)
Figure 5: Ablation for generative coherence, latent classification accuracy based on the private en- codings, and generative quality, as a function of private latent dimensionality. The percentages on the x-axis indicate the relative size of any private subspace with respect to the shared subspace.
modalities, at the expense of carefully constraining the size of crodality-specieee subspaces. This however has two undesirable effects. On one hand inevitably limits modelling capacity for private information, which hampers generative quality, as showcased in figure [c] On the other hand, and more importantly, achieving an adequate balance between the two performance criteria heavily de- pends hyperparameter tuning, which calls for expensive model selection procedures in practical settings.
In contrast to the alternative approaches, the MMVAE+ proves be significantly more robust with respect to hyperparameter values controlling the size of the modality-specifie subspaces. In particu- lar, our approach does not incur in a shorteut. Instead shows better performance, jointly looking at generative quality and generative coherence, when shared and private latent subspaces for each modality have comparable capacities. Combining these results with the ones from the previous sec- tion, it is evident how our approach successfully exploits having additional modality-specif latent codes to enhance generative quality, without compromising semantic coherence.
4.3 CUB SMAGE-CAPTION
In this section, test the MMVAE+ on the CUB Image-Captions Shi al. 2019 Netzer al. 2011) dataset, and compare its performance with existing approaches in terms of conditional and unconditional generation, in order validate our approach and test its impact on complex real- world dataset. For the conditional caption-to-image generation (Figure 6). we again observe that modality-specific information tends collapse to average values for mixture-based models, namely MMVAE, MoPoE-VAE and mmJSD, whereas MVAE and MVTCAE show convincing generative quality but severe lack of coherence (cp. Daunhawer et al., 2022). Similar conclusions can be drawn for conditional image-to-caption generation (Figure 7 In contrast, MMVAE+ shows signif- generation. Specifically, our
icantly better results for both caption-to-image and image-to-captie model achieves high sample diversity for both the image and text modality. Hence, we demonstrate the effectiveness of our approach for more complex real-world data, where there is sizeable amount of podality-speciff information and significant heterogeneity across modalities.
Conditional coherence
Conditional FID
MVAE
0.271 (é60007)
172.21 (#:39.1)
МVTСAE
0.221 (#0.007)
208.43 (#1.10)
mmJSD
0.556 (40.118)
262.80 (#6.98)
MMVAE
0.713 (57.057)
232.20 (#2.14)
MoPoE-VAE
0.579 (20.118)
265.55 (#4.01)
MMVAE+ (ours)
0.721 (4.0.00 (000900)
164.94 (4:1500
Table Comparison of generative quality (in terms FID, lower better) and generative coherence (higher is better) for conditional caption-to-image generation the CUB Image-Caption dataset.
8

---

Published as a conference paper at ICLR 2023
thes bird hex and and and.
this bard has yerlow with yellow yellow
this bidd has yeloow yellowe, yoliowr yelliwr black.
bird with bright yellow buily track
mouuly brown with white breast and tan
palo brown and wing and tlue ball.
black
and black
with
this aerd has wings white has his small orange neck.
thes bird has black and has and and.
bird viligg that are brown and has baily.
this small bird bind beily a and and
bird
rounded threet and curved
verry pointed
ball
dark brown
coverts and and and wingee
bird white teps. tht head wide
this bird with yellow tiny threat yellow eeck, parple.
this bard has with while
this bled hos with the with and and hes
this medium Net hidd bid head whihe,. and and and and.
winge that
and. and.
with
any and hos
beily.
(a) MVAE
(b) MVTCAE
(c) mmJSD
(d) MoPoE-VAL VAE
(e) MMVAE
(1) MMVAE+
Figure 6: Conditional image-to-caption generation on the CUB
dataset.
thes bird
thes bird Maach, and Llke in bleck beak.
This ahirry black, and
this bird
shiny brock, and Blue
shiry black, and blook
ohiny back, and color with brack
truck, like in black beak.
color, with
black beak.
black beek.
beok
(a) MVAE
(b) MVTCAE
(c) mmJSD
(d) MoPoE-VAE
(c) MMVAE
MMVAE+
Figure 7:
Conditional caption-to-image generation on the CUB Image-Caption dataset. For each
model, the input caption used conditionally generate four images.
To quantitatively evaluate coherence, we design new coherence proxy for the CUB dataset. While for the PolyMNIST dataset, the information shared between modalities always con- sists of the digit label, for the CUB Image-Captions dataset we do not have annotation for the shared content between modalities. Further, the ResNet-feature evaluation used in previous works (Shi) jet al. 2019 202T> not suitable to evaluate models that were trained on real images. Therefore, we design a coherence proxy for conditional caption-to-image generation based on the color of the described bird (see E.2 for details). Based on this metric, providee a quantitative comparison of the models for caption-to -o-ione---- generation in terms of generative coherence and generative qual- ity. We again quantify visual quality of the produced images using the FID score. The quantitative results suggest that the MMVAF+ can model complex real-world datasets, where the predominant models fail to achieve satisfying results in at least one of the two performance criteria.
5 CONCLUSION
This work introduced MMVAE+, a new model in the family of multimodal VAEs, which derives from the MMVAE and considerably improves its generative quality without compromising semantic coherence. The proposed model uses separate shared and modality-specific subspaces and leverages auxiliary priors to facilitate the estimation of cross-modal reconstructions. We show that the model optimizes valid multimodal ELBO and that it effectively encodes shared and modality in- formation in the respective subspaces. Our approach can be generalized and extended to other types of decompositions of the joint encoder, such as the mixture-of- -roducctsssssssooooooooo of experts. We leave this as possibility for future work. On both synthetic and real-world data, we demonstrate the effectiveness of our approach for enhancing generative quality without compromising semantic coherence. Com- pared to previous approaches with separate latent subspaces, MMVAE+ is significantly more robust to hyperparameter values controlling the capacity of modality-specific subspaces. As such, MM- VAÊ+ takes an important step towards the applicability of multimodal VAEs to complex, real-world multimodal datasets.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
Emanuele Palumbo was supported by the grant #2021-911 of the Strategic Focal Area "Personal- ized Health and Related Technologies (PHRT)" of the ETH Domain (Swiss Federal Institutes of Technology). ID was supported by the SNSF grant #200021-188466. Special thanks to Carl Allen, Thomas Sutter and Yarden As for helpful discussions and/or feedback on the manuscript.
ETHICS STATEMENT
Given the nature of the experiments, and the datasets used no ethical concerns regarding this work.
REPRODUCIBILITY STATEMENT
We provide details about datasets, metrics, hyperparameters and architectures in the Appendix, which can aid reproducibility our method along with the provided code.
REFERENCES
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. Proceedings of the AAAI Con- ference on Artificial Intelligence, 2018.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung- Woo Ha, Sunghun Kim, and Jacgul Choo. Star- gan: Unified generative adversarial networks for multi-domain image-to-image translation. In Conference on Computer Vision and Pattern Recognition, 2018
Imant Daunhawer, Thomas M. Sutter, Kieran Chin-Cheong, Emanuele Palumbo, and Julia E Vogt. On the limitations of multimodal VAEs. In International Conference on Learning Representa- tions, 2022.
Reuben Dorent, Samuel Joutard, Mare Modat, Sébastien Ourselin, and Tom Vercauteren. Hetero- modal variational encoder-decoder for joint modality completion and segmentation. Medical Image Computing and Computer Assisted Intervention, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems, 2017.
Irina Higgins, Loie Matthey, Arca Pal, Christopher P. Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representattinns 2017.
HyeongJoo Hwang, Geon-Hyeong Kim, Seunghoon Hong, and Kee-Eung Kim. Multi-view repre- sentation learning via total correlation objective. In Advances in Neural Information Processing Systems, 2021.
Tom Joy, Yuge Shi, Philip Torr, Tom Rainforth, Sebastian M Schmon, and Siddharth N. Learning multimodal VAES through mutual supervision. In International Conference on Learning Repre- sentations, 2022.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations, 2014.
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Changhee Lee and Mihacla van der Schaar. A variational information bottleneck approach to multi- omics data integration. In International Conference on Artificial Intelligence and Statistics, 2021.
10

---

Published as a conference paper at ICLR 2023
Mihee Lee and Vladimir Pavlovic. Private-shared disentangled multimodal vae for learning of latent representations. In CVPR Workshop on Multimodal Learning and Applications, 2021.
Kodai Minoura, Ko Abe, Hyunha Nam, Hiroyoshi Nishikawa, and Teppei Shimamura. A mixture- of-experts deep generative model for integrated analysis of single-cell multiomics data. Cell Reports Methods, 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Read- ing digits in natural images with unsupervised feature learning. In NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International Conference on Machine Learning, 2016.
Maximilian Seitzer. pytorch-fid: FID Score for PyTorch. hattps///gt.ubb..commmeee.s fitorch-fid, August 2020. Version 0.2.1.
Yuge Shi, N. Siddharth, Brooks Paige, and Philip Torr. Variational mixture-of-cxpert. autoencoders for multi-modal deep generative models. In Advances in Neural Information Processing Systems, 2019.
Yuge Shi. Brooks Paige, Philip Torr, and N. Siddharth. Relating by contrasting: A data-efficient framework for multimodal generative models. In International Conference on Learning Repre- sentations, 2021.
Thomas M. Sutter and Julia E. Vogt. Multimodal relational VAE. NeurIPS Workshop on Bayesian Deep Learning, 2021.
Thomas M. Sutter, Imant Daunhawer, and Julia E. Vogt. Multimodal generative learning utilizing jensen-shannon-direegene In Advances in Neural Information Processing Systems, 2020.
Thomas M. Sutter, Imant Daunhawer, and Julia E Vogt. Generalized multimodal ELBO. In Inter- national Conference on Learning Representations, 2021.
Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Joint multimodal learning with deep generative models. arXiv preprint /6110/// 2016.
Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhut- dinov. Learning factorized multimodal representations. In International Conference on Learning Representations, 2019.
George Tucker, Dieterich Lawson, Shixiang Gu, and Chris Maddison. Doubly reparameterized gradient estimators for monte carlo objectives. In International Conference Learning Repre- sentations, 2019.
Miguel Vasco, Hang Yin, Francisco S. Melo, and Ana Paiva. Leveraging hierarchy in multimodal generative models for effective cross-modality inference. Neural Networks, 2022.
Ramakrishna Vedantam, Ian Fischer, Jonathan Huang, and Kevin Murphy. Generative models of
visually grounded imagination. In International Conference on Learning Representations, 2018 88.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech- UCSD Birds-200-2011 Dataset. Technical report, California Institute of Technology, 2011.
Weiran Wang, Xinchen Yan, Honglak Lee, and Karen Livescu. Deep variational canonical correla- tion analysis. arXiv preprint 100.04161. 2016.
Jannik Wolff, Tassilo Klein, Moin Nabi, Rahul Krishnan, and Shinichi Nakajima. Mixture-of- experts vaes can disregard variation in surjective multimodal data. In NeurlPS Workshop on Bayesian Deep Learning, 2021.
Jannik Wolff, Rahul Krishnan, Lukas Ruff, Jan Nikolas Morshuis, Tassilo Klein, Shinichi Naka- jima, and Moin Nabi. Hierarchical multimodal variational autoencoders, 2022.
11

---

Published as a conference paper at ICLR 2023
Mike Wu and Noah Goodman. Multimodal generative models for scalable ing. In Advances in Neural Information Processing Systems, 2018.
learn-
Jun Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In International Conference on Computer Vision, 2017.
12

---

Published as a conference paper at ICLR 2023
A PROOFS
Lemma 1
The MMVAE+ objective (Equation 5 is a valid lower bound on logpe (X).
Proof. We assume the data generative process
M
pe p 2, W) p(2) II po... (2ml wm wm () m==1
where z encodes the shared content between modalities, and each wm encodes modality-specific information for modality Note that shared latent information in and modality-speciff infor- mation for each modality wi..... WM =: W are assumed to be independent. We start by approxi- mating posterior inference for the shared latent space with coatture-of-expeeee variational encoder q4 (=|
2mm Udmm (z)am)) With our assumptions the objective
M
p(=)po... (2 Inzrr m po,, (1 (2)
CMMVAE(E1
M
E
Roam (=/mm)
log
40 (=|X)
(6)
m==1
is a valid lower bound on the log-evidence log pe (X).
Note that this the MMVAE objective (Shi al. 2019), where for each term in the sum the en- coding z ~ T (z))m) ) is sampled from a unimodal encoder, and used to compute the conditional likelihood of all'M M modalities: in particular both self-reconstruuttinn likelihood log Po... (m |=] and cross-modal reconstruction likelihoods log Pe,, (z,, x²) for & are evaluated. We derive the MM- VAE+ objective by adopting different estimators for the two kinds likelihood terms. In detail, we estimate log, pe... (Imm 22) using variational encoder 4dwm (wm mm) for modality-speciie informa-
tion and deriving the lower bound
p6 (Im Wm 2 2) ( Wmn)
log Pe... (Im xx) 2 E Rown (wm. (m)) log
gowm
(Wm [mm)
In contrast for cross-modal reconstruction likelihoods we propose the estimator
log pa.. (zx,/z) log -,, (w,, Pe,, (x,, |2, w. E,,,,,,,,, (mn,) log pe,, (I,, u.
(8)
using auxiliary prior distribution I' (w,) specific to each target modality. Note the first step holds by definition of conditional expectation, while the second step follows from Jensen's 's inequality.
Plugging the derived expressions in and equation we recover the MMVAE+ objective
LNMMAEE (#1:M) =
1
Σ
E
ae (x (e))) log
pe.. (xm 22, Wm (p())p((w,))))
II
po., (arn (z, wa)
M
Gewmn (W0 (mm) qv (2 ))4mmm
(wm Dm)
m=1
{to,, nurt (w., magt
From the fact that equation 6 an ELBO. and the fact that the derived expressions and are lower bounds, we conclude that the MMVAE+ objective is an ELBO.
B ADDITIONAL THEORETICAL RESULTS
Here we prove lemma, which formalizes the fact that with the MMVAE+ objective cross-modal re- constructions affect only the shared subspace but not the modality-specific subspaces. Specifically, in the MMVAE+ objective (Equation 5), the encoding z ~ qø.... (z| is used for the reconstruc- tion of modality m, as well as for cross-modal reconstruction consequence, the gradients for parameters dz.,..... ΦzN, depend on the reconstruction likelihood of all modalities. Consequently. encoders {q0.m (z (m ( m==1....M are optimized for to contain information useful to reconstruct all modalities, i.e., shared information. In contrast, the encod- ing wnn ~ 40 (w, (z,)) is used only to compute the reconstruction likelihood of modality m. Hence, gradients for encoder parameters &w > ........ Pwar only depend self-reconstruutiinn terms.
the remaining modalities. As a
13

---

Published as a conference paper at ICLR 2023
As result, for optimized cross- moosl reconstruction, all shared information needs to be contained in and predicted from 2. On the other hand, private information for modality m can be modelled consequence of optimizing self-re self-reconstruction. As confirmed by our experimental results, our proposed objective leads to shared and private information being encoded in separate shared and modality-speciff latent subspaces respectively.
in wm. 90w,, (w (m
Lemma 2. Computed gradients for the MMVAE+ objective with respect to private encoder param- eters ow, Own do not depend on rross-modal reconstruction likelihood terms, while gradients with respect to shared encoder parameters ..1...., 9ZM depend on the reconstruction likelihood of all modalities.
Proof. Computing the gradients for the MMVAE+ objective (Equation 5) with respect the private encoder network parameters ow.. for a given modality m {I..... M gives
pm.. (3 12, Wm (p
frem CNMMVAEL(D1:M) (FI:M)
log
gexm (xs (e))) Adiun (10m (m
@own (Wm F
Pommsem Brrm (3m)
owo
Note that the resulting function does not depend decoder parameters {0,, h i.e. does not depend on cross-modal reconstruction likelihoods. Computing gradients with respect the shared encoder network parameters dz... for given modality E ( JM instead gives
LMMIAEE (F1:M)
=
dem
do (x((e)) qaur (w.. [m (1i Now (w hapte
log pm (Ie Iz, wm)p(z) II nottmm
po., (xc, 12, 2.,))
E Z2 (z/or,
log
M
qa (z X))
V dxm
Ge. drem (X)
which is function of the reconstruction likelihood of all modalities given inference from modality m, through decoders parameterized by θ
1 }
C CHOICES OF AUXILIARY PRIORS
In this section we elaborate on the role of auxiliary prior distributions r1 (1(1),, ..., Far (wxs in our proposed objective, and the specific design choices for such distributions.
In our experiments, auxiliary priors we choose rero-mean distributions with variance parameter that is learnt at training time. Notably, the learnt variance is separate parameter vector (for each private subspace) that does not affect the encoders. Moreover, variance parameter, and hence the (w)), T' (wM) distributions not computed conditioning any specific data sample. Therefore having a variance parameter does not affect our results from lemma]
Having learnt variance is key component as it allows the auxiliary priors to adapt the effective value range of the private encoding for cross cross-modes modal reconstructions. Specifically, the prior can adapt indicator that informs the decoder about selftreconssttuctions versus
the value range cross cross-mode modal reconstructions. Empirically, we find support for this hypothesis, as each auxiliary prior tends to become sharp distribution (i.e., approximately delta function) in the course of training.
14

---

Published as a conference paper at ICLR 2023
D INCLUDING A MULTI-SAMPLE ESTIMATOR AND A & HYPERPARAMETER IN THE MMVAE AND MMVAE+ OBJECTIVES
D.I MULTI-SAMPLE MMVAE AND MMVAE+ OBJECTIVES
To tighten the bound the log- log-evidence in estimator. In fact the K -ssaple objective
Shi al. (2019)
propose to multi-sample
M
1 K
K
pel X 2*) X)
C MMVAE (III.)
Σ 12
Bg mom (2/mm) log
M
To
tighter ELBO compared (2) for K > 1, it can be shown with Jensen's 's inequality. However, as multi-sample estimators can lead to undesirably high variance, Tucker et al. 20199 have proposed doubly-reparammelerieed gradient estimator (DReG) reduce the variance in estimated gradients for latent-variable models with multi-sample objectives. [Shi et al. (2019) resort to this estimator in computing gradients for the multi-sample version of the MMVAE objective.
Analogously for the MMVAE+ objective, a tighter bound on the log-evidence can be enforced if one uses mulli-sample estimator. In fact the K K-sample version of the objective in (5)
M
K
Cms MMVAEE (xI..) = WV
M
Σ
E
m mgddo (2/ar...
log
K
Ca (X, =* "0, ........ w
WM
goaeen (20m (20m) TT, (20 (aponiinnn Iusto
(10)
with
w w = Pd.. (Imm 12 w (mmpp(( 2 (w
Co.e(X, xt, w
Pa.. (x 20
qo (x { gep..... (w [m
a tighter ELBO compared to 5 for K > it can be shown with Jensen's inequality,
D.2 BALANCING RECONSTRUCTION AND REGULARIZATION WITH A ,8 HYPERPARAMETER
As common practice in the VAE literature, the objective in ① can be rewritten as sum of recon- struction term and KL-divergence term
L (xI.M) = Eg (=|x) [log X x]] +Dxx(90(x
|| p(=))
(1I)
where a 3 hyperparameter weighting the latter term is introduced to control smoothness of the latent space (Higgins al. 2017). With the assumption of a mixture-of offeeerrss joint encoder, one can in the same way rewrite the MMVAE objective as the sum of reconstruction and KL-divergence term, weighted with a B hyperparameter
M
LMMVAR (II:M)
Σ
Ege. (=|X)
log logpel pe (ogge(X11)) (X1z)
-33 [ p(z))
M
m=l
M
pe (x (X
(x|))p((z))*
Σ m=1
Eggn (=|
log
(12)
M
(24(2
From the last expression, derive multi- sample version of the MMVAE objective
M
pe pe(X > (p(=)))
Cnns MMVAR (X1:As)=
M
1:K moo. (=(am))
log
(go (z* X))
(13)
where the hyperparameter is introduced. Note that Jensen's 's inequality proves [1 is a tighter bound the log-evidence compared to for K > 1 and any fixed value of 8 >
15

---

Published as a conference paper at ICLR 2023
Analogously, we can derive 3 B-weighted version of the K K-sample MMVAE+ objective
M
pmm. (Z Wmn (p (pppp(/p(((p)))
L (#I.M)
M
soam (#)
log
(00, (2) (x|X)) 19 (Wm (4 ))3
II nyftom
pon. (mn (z, (nn
Serenn (1s (20m) (in woth (aF,.) Jupte
and the corresponding -samll objective
C + (I1:M)
C
1 M
Σ
E
= -u (x) (m))
log
K
Σ
D e.a e (x, "0, w m ....... wxx)
an 1i Iohmm (wimd (mm))
k=1
{ 1-8 ~in (w0 tagemm
with
po.. (2:ml 12t with (p()))((())))) pp( )) (go. (z X ))dwo ( 3m) )
(X 2", "........
w
wM MM
II pe.. (2x,1z", in mot
Note that in our experiments, when training the MMVAE and the MMVAE+ with the respective K s-sample 3-weighted objectives we make use of the DReG estimator (Tucker al. 2019).
E TECHNICAL DETAILS FOR DATASETS AND METRICS
E. DATASETS
In this work, we report experimental results on two multimodal datasets introduced in previous works, namely PolyMNIST (Sutter et al.) (022)) and CUB Srage-Captions (Shi et 2019: Netzer
et al. 2011).
MNIST synthetic dataset consisting of five image modalities. Each datapoint consists of five images depicting MNIST samples sharing the digit label, patched on random crops from five different background images, one for each modality. Shared information between modalities digit label, while private information comprises the background and the style of the handwriting, since MNIST digits are shuffled prior to the pairing.
The Caltech Birds (CUB) Captions dataset consists of images of birds paired with matching linguistic descriptions. Making this experiment ambitious is the high amount of modality-specific information for each modality, due to the different nature of the information sources. Moreover, while for PoISMNIST the amount of shared information for each sample is fixed, always consisting of the digit label, here the amount of shared information present In fact, descriptions can be more or less detailed, leading to more or less shared information between modalities. This dataset presents lot of challenges typical of realistic experiment, which makes it relevant setting to benchmark existing approaches.
vary from datapoint to datapoint.
2
Z
M
3
4
S
6 7 8
9
2
3
4
7
thö is small, whrtsh
this purticulur hind belly that vellow I..]
a ammal bind oronge body
this is buaaell white aulloo
this bird ia bluck with whito
4
5
8
q
with
2
5
ding
Figure 8: Illustrative samples from PolyMNIST (left) and CUB Image-Caption (right) respectively. Images taken from Shi et al. (2019) and Daunhawer et al. (2022).
E.2 METRICS
our experiments, to evaluate generative quality for image modalities we adopt the FID score (Heusel et al. 2017), state-of-the-ard metric to quantify visual sample quality for gener- ative models in image domains, which has been shown to correlate well with human judgement. To compute FID scores use the implementation from Seitzer (2020).
FID score
16

---

Published as a conference paper at ICLR 2023
white
yellow
blue [90, 50, 70] [158, 255, 255] brown [24, 255, 255]
red
[0, 0, 120]
[25, 50, 70] [35, 255, 255]
[0. 50, 70]
[159, 50, 70] [180, 255, 255]
[180, 18, 255]
[15, 255, 255]
green [36, 50, 70] (89, 255, 255]
gray [0, 0, 50]
black
[0, 0, 0]
[180, 18, 120]
[16, 50, 70]
[180, 255, 50]
Table 2: HSV (Hue, Saturation, Value) color ranges used to classify pixels in color classes. Each pixel in HSV color coding is represented by three values. Lower and upper limit(()) for each color class range(s) are reported below each color class label. Note that include all tonalities of red one has to consider two distinct ranges.
Generative coherence: PolyMNIST To compute generative coherence in the PolyMNIST ex- periment, as done in previous works (Shi et al. 2019 Sutter et al.) 2021 Daunhawer et al. 2022). we use the training samples for each single modality to train a classifier for the digit label. The resulting classifiers are then used to obtain quantitative measure for conditional and unconditional generative coherence. To measure conditional generative coherence from a given input modality m given target modality n, we feed the conditionally generated sample to the classifier for the target modality n, then compare the predicted digit label with the true digit label. The matching rate for the whole test set measures conditional generative coherence from modality m to modality n. To obtain single representative metric for PolyMNIST, we compute the average coherence for each target modality, and then average the five obtained results. For unconditional generation, co- herence is measured by feeding jointly-generater trained classifier. Then the measure for coherence the rate with which all classifiers output the same predicted digit label on the total number of generations.
samples for each modality to the corresponding
Generative coherence: CUB Image-Captions To have a quantitative assessment of semantic co- herence in the CUB Twage-Captions experiment design novel generative coherence metric for caption-to-image generation. In detail, we construct eight captions of the form this bird is completely [color]. where [color] takes values in the set (white, yellow, red, blue, green, grey, brown, black), ). We divide the HSV color coding range according to these eight color classes (see Table Then, for each of the starting captions we generate ten images. For each generated image we count the pixels belonging to each color class. We label an image coherent if the color for the starting caption is among the two color classes with the highest pixel count in the image. We take into account the two color classes with the highest count, as the highest count for some images might be the background color rather than the color of the depicted bird. Finally, the ratio of coherent images over the total number of generated images is our coherence metric.
We want to point out that, while Shi et al. (2019) propose a quantitative metric for the CUB Image- Captions dataset, this metric is effective only when dealing with the simpler version of this exper- iment that uses pretrained ResNet-features, rather than training on actual images. [Shi et al. (2019) train the model ResNet embeddings rather than real images: for image generation the model actually generates an embedding, that is matched with the nearest-neighbour embedding among all embeddings of the images in the original data. Therefore what the model outputs in an image be- longing to the test set. This results in having high-quality images as output, but no generation in pixel space happens, which makes the task easier. The coherence metric used by [Shi al. (2019) relies on the fact that generated images are essentially taken from the dataset itself, and therefore come from the same distribution as the data. So, cross- gesss-geneetted images can be fed to the pre-trained ResNet used at training to compute meaningful embedding to be compared with the embedding for the starting image. In our case setting instead, images are generated directly in pixel space, which means there would be distribution shift between the generated images and the starting images (c.g. generated images are blurrier). With such a distribution shift, feeding generated images to the ResNet encoders trained original data does not yield meaninful results, which invalidates such metrie for our setting.
Latent classification accuracy In the PolyMNIST setting, where have annotations for the shared information between modalities, i.c. the digit label, we use latent classification accuracy (Sutter al. 2021 Daunhawer et al. 2022 Shi et al. 2019) proxy for the amount of shared
17

---

Published as a conference paper at ICLR 2023
information encoded in given latent subspace. To compute this metric, digit-classifier is trained on the latent embeddings and its accuracy on the test set is a proxy for how much information about the shared digit content is encoded in the latent code. For our results in section 4.2 where we aim to have a measure of how much shared information is present in modality-speciffe subspaces, we train digit classifier on each of the five coddality-speciffe latent embeddings, and then average the test accuracies of the five classifiers to obtain a single metric.
F TECHNICAL DETAILS FOR THE EXPERIMENTS
PolyMNIST (results in Section 4.1 and Appendix G Following the suggestions by the authors in Shi et al. (019)) to reach best performance on image-to-image datasets, we train the MMVAE assuming Laplace priors. likelihoods and posteriors, constraining their scaling across the D dimen- sions to sum to D. For compatibility, we use the same settings for the MMVAE+. In our experiments we set the size of the shared latent subspace for MMVAE+ to 32 dimensions. In order not to induct any bias, and have equal shared and private latent capacity for each modality, we also set the size of each mopelity- latent subspace to 32 dimensions. For a fair comparison with the MMVAE, we set the size of the latent space to be equal to the total latent capacity of MMVAE+, namely 160 dimensions. When training both models with a multi-sample estimator in the objective, we reduce the size of the latent space of VAE to 64 dimensions, due to main memory constraints. MM- VAE and MMVAE+ are trained for 50 epochs when trained the K sample version the respective objectives, using K 10 samples, while they are trained for 150 epochs when trained without multi-sample estimators. We train the mmJSD model again using a latent space of 160 dimensions, and for 150 epochs.
For MVAE, MoPoE- VAE, and MVTCAE, we use the settings in the work Df et al. (2022) and Hwang et al. 2,02T, in order obtain best performance in this experimental setting. We assume Laplace likelihoods and resort to Gaussian priors and posteriors which enable to compute product distributions in closed form. We train both MVAE and MoPEE-VAE for 500 epochs with latent space of size 512, while MVTCAE (Hwang et al. 2021 is trained for 300 epochs, with the same latent space size.
We compare all models for range of representative values of the regularization hyperparameter E {1.0, 2.5, 5.0). Due numerical instabilities the MVAE could not be trained with = 5.0, and therefore for this model we set 3 = 3.0 as the last value of the regularization hyperparameter. Finally, for all compared models we use the same ResNet encoder and decoder networks, for each modality.
PolyMNIST (results in Section 4.2 For a fair comparison between the models, we train the MM- VAE using the objective in 4 the mmJSD with the objective assuming factorized latent space (Sutter et al. 2020), and the MMVAE+. using the same ResNet encoder and decoder networks for each modality, assuming Laplace likelihoods, and training all models for 150 epochs, without resorting to multi-sample estimators for any of the models. Note that the size of the shared latent subspace is always fixed 32 dimensions.
CUB Image-Captions On this experiment, for all compared models, we assume Laplace and one- hot categorical likelihood distributions for images and captions respectively, with Gaussian priors and posteriors. and set the latent space capacity to 64 dimensions. The size of private latent sub- spaces for MMVAA+ chosen so that, for both modalities, summing the size of the private subspace and the shared subspace also amounts to 64 dimensions. We train MVAE, MoPoE-VAn VAE, MVTCAE and mmJSD on this experiment for 150 epochs, with the settings to replicate best performance used in Daunhawer et al. (2022). In particular, ResNet encoder and decoder networks are used for both image and text modalities. MMVAF and MMVAF+ are trained for 50 epochs with the K- K-sample version of the respective objectives and K = 10, using the DReG estimator. Both models use Resnet network architectures for the image modality and CNNs for the text modality.
General specifics for the experiments All models are trained using the Adam optimizer (Kingmal & Ba 2014h. We choose a learning rate of 5e-4 for MVAE and MoPoE- VAE, following Daunhawer et al. 2022). while for MMVAE and MMVAE+ we choose 1e-3. mmJSD, and MVTCAE are also trained with learning rate 1e-3, as suggested in the respective implementations by the authors. All quantitative results are averaged over three seeds, and reported with standard deviations. Finally,
18

---

Published as a conference paper at ICLR 2023
want to specify that when comparing unconditional coherence in both experiments, for approach which has separate latent subspaces, we sample the shared latent z from its prior p(z), and the modality latent wm for each modality from the respective prior p( p(wm).
G ADDITIONAL RESULTS
G.1 POLYMNIST
Detailed quantitative model comparison In 3able we report the numerical values for each met- ric the quantitative comparison in Section 4.1 These results show that only the MMVAE+ reaches high generative quality and high semantic coherence across different 3 values, for both conditional and unconditional generation. Alternative models instead markedly underperform in one of the two criteria, or achieve convincing scores only for conditional or unconditional generation. The latter is the case with MVTCAE. fact, while being an important improvement over the MVAE for condi- tional generation with greatly superior coherence, its performance markedly drops for unconditional generation, with extremely low values for coherence.
Conditional generation In figures and TO we report additional and more extensive qualitative results for the comparison between MMVAE+, MMVAE, MVAE, MVTCAE, mmJSD and MoPoE- VAE in section 4.1 for conditional generation.
Latent classification accuracy While conditional generation results showcased in Section 4.1 al- ready offer sufficient evidence in this regard, here show yet other results supporting the validity of our modelling assumptions where the shared latent subspace and the private latent subspace con- tain disentangled shared and modality-spente we use latent classification accuracy (see Appendix F.2) to prove that the MMVAF+ achieves fac- torization between shared information in and private information in w1:-.... WM For ablation, similarly to section 4.2] we vary the proportion of latent dimensions allocated to the shared and the private subspace: with the shared subspace fixed to 32 dimensions, we vary the number of dimen- sions for each private subspace between 4, 8, 16, 24, 32. Hence we go from having 1 12.5% of the shared latent capacity allocated the wm subspace for each modality m, to up to the two subspaces having the same size. The showcased results in Figure 12 indicate that the subspace successfully encodes shared content across modalities, consistently throughout modelling assumptions for MMVAE+, showing they are robust to variation in hyperparameters. The accuracies obtained with classifiers trained on private embeddings are, as expected, markedly low. These results offer additional empirical evidence demonstrating that MMVAE+ learns repre- sentations that factorize into shared and private subspaces consistently with respect to changes in hyperparameters.
information respectively for each modality. In detail,
ablation, which validates our
Additional baseline results As an additional baseline to compare our approach with, we include results obtained having the same assumptions for the generative model in section[.22) but assum- ing a proeeettof-experts joint encoder. The resulting objective essentially extends the MVAE (Wul & Goodman 2018) have separated shared and modality-speciiff latent subspaces. The optimized objective is
pel X 2, W)
S (X) E
go (=) gown: (w) (11)
log
go. (z X)) Th aowm (wi [mm)
(14)
Yowar (wo (mar))
where
M II
q0. (z|X)
40am (2(2m))
As is expected and can be seen in Figure this additional baseline model also proves to suffer from the shortcut problem, as the other existing multimodal VAEs with separate shared and modality- specific latent subspaces.
19

---

Published as a conference paper at ICLR 2023
1.
150
0.8
Generaaive qualty
100+
0.5
0.6
lamerot
Sund ensumi in
Joursos
12.5%
2006
5006
79%
100%
12.9%
2555
507%
7505
100%
12.55
25%
71566
100%
Mudality-specilff lasent capacity
Condality-specciin capucity
Modality specific latent cupacity
(a) Generative coherence (higher better)
(b) Latent classification accuracy from modalit--specific encodings
(c) Generative quality (in terms of
FID score, lower is better)
(lower is better)
Figure 9: Results for the ablation in Section 4.2 including the factorized version of MVAE (Wu & Goodman 2018) an additional baseline.
MMVAE
2
2
2
4
3 7
7
MMVAB+
Figure 10: Conditional generation qualitative results for MMVAE and MMVAE+ the PolyM- NIST dataset. For each different starting modality, we show input samples in the top row, and four instances of conditional generation for the remaining target modalities.
20

---

Published as a conference paper at ICLR 2023
R
D
2
&
O
O
COS
OODAA AU
MVTCAE
mmnSDD
8
7
MoPoE-VAE
7 M
216 7
0
W G
8
O
7 9
2
Figure 11: Conditional generation qualitative results for MVAE, MVTCAE, mmJSD and MoPoE- VAE on the PolYMNIST dataset. For each different starting modality, we show input samples in the top row, and four instances of conditional generation for the remaining target modalities.
21

---

Published as a conference paper at ICLR 2023
1.0
Latent classification accumicy
0.8
0.6
0.4
0.2
0.0
12.5%
25%
50%
75%
100%
Modality-speciiff latent capacity
Figure 12: MMVAE+: latent classification accuracy computed on the shared latent embedding and the poodality-speciffe latent embeddings W = Wi..... WM. as a function of relative capacity of modality mppelity-specifie subspaces. Note that this result validates approach, which consistently encodes shared information across modalities in 2, while this information is not present in W.
1.0
Unconditional
Conditional
FID
Coherence
FID
Coherence
MVAE
50.65 (20.72)
0.007 (#0.001)
82.59 (20)22)
0.093 (=00000)
MVTСAЕ
110.85 (#2.61)
0.000 (#0.000)
58.98 (20.62)
0.509 (=00000)
MDJSD
179.76
(1:2.77)
0.054 (10.011)
209.98
(1:1.26)
0.785 (::0.023)
MoPoE-VAL VAB
98.56 (1:1.32)
0.037 (1:0002)
160.29
(1:4.12)
0.723 (::0.006)
MMVAE
165.17
(2:3400 40)
0.222 (10.019)
152.11
(1:4.11)
0.837 (::0.004)
MMVAE(K=10)
158.95
(91.79)
0.292 (E0.016)
196.62 (+1.99)
0.865 (=:00006)
MMVAE+
86.64
(+11004
0.095 (E0.020)
80.75
(#0.118)
796 (=0.010)
MMVAE+ (K=10)
95.81 (d:4.04)
0.244 (#0.023)
96. 16.11 (25.66)
0.859 (20.012)
= 2.5
Unconditional
Conditional
FID
Coherence
FID
Coherence
MVAE
58.53 (2))
0.080 (20.000)
85.23 (#9.37)
0.298 (d:0.04))
MVTСAЕ
87.07 (1:0.9)
0.003 (200 0.0000
62.55 (11)30)
0.591 (1:0004)
mmJSD
180.55
(2:667))
0.060 (20.010)
222.09 (1:.34)
0.778 (1:0003)
MoPoE-VAs VAE
107.11
(+0.780)
0.141 (+0.005)
178, 27 (+2.01)
0.720 (4900088)
MMVAE
164.71 (4:3.17)
0,232 (+0.010)
150.83 (422.99)
0.844 (4-0.010)
MMVAE (K=10)
164.64 (#2.84)
0.379 (:0 (#0.034)
197. 24 (#1.87)
0.853 (#0.007)
MMVAE+
96.01 (2:2.10)
0.344 (20.013)
92.81 (#0.78)
0.869 (#0.013)
MMVAE+ (K=10)
103.67 (#2.22)
0.446 (#0.000)
102.60 (22.17)
0.900 (#0.002)
8 = 5.0
Unconditional FID 61.25 (#0.40)
Conditional
Coherence
FID
Coherence
MVAE
0.112 (20.010)
90.37 (#3.22) 20)
0.301 (20.022)
MVTСAE
85.43 (#2.80)
0.029 (20.001)
74.61 (#3.41)
0.604 (20, 004)
mmJSD
186.49
(1:2.99)
0.076 (10.018)
226. 20 (1:2991)
0.784 (200009))
MoPoE-VAE
122.68
(1:1.66)
0.238 (10.001)
182.99
1.96)
0.673 (::0.002)
MMVAE
164.29
(#2.97)
0.229 (t0.017)
152.11 (#3.18)
0.839 (20.010)
MMVAE (K=10)
176.79 (+1.32)
0.407 (#0.022)
193.79
(82)88)
0.817 (20.016)
MMVAE+
109.08
(E1.41)
0.421 (20.006)
107.78 (20.88)
0.8365 (10.022 (2.)
MMVAE+ (K=10)
115.57 (#2.22)
0.501 (20.004)
118.30 1.42)
0.868 (=0.005)
Table 3: Comparison of model performance for generative quality (in terms of FID, lower is better) and generative coherence (higher better). Both conditional (cross-modal) generation and uncondi- tional generation performance are evaluated. Each table shows the results for the compared models for a different 3 value. Note that values the table relative to 5.0 for MVAE are actually computed with 3 = 3.0 due to numerical instabilities.
22

---

Published as a conference paper at ICLR 2023
G.2 CUB MAGE-CAPPIONS
Image quality for generation In figures 13 and we report unconditional and conditional image generation results for the compared models on the CUB Image-Captions dataset. Image generation results for MMVAE, mmJSD, and MoPoE-VAE in this complex experiment are characterized by the presence considerable number of generated samples for which modality-specife features are collapsed to average values, resulting in blurred images. MMVAE+ results on the contrary not exhibit signs of this pattern, which indicates good modelling of modality-specific information even in this complex setting, compared to the other mixture- based approaches. For the MVAE and MVT- CAE, product-basee approaches, we witness instead the presence of significant variation in produced images, but low generative coherence comparing with our approach. figure [6 we also show the generated samples resulting from computation of the coherence score for this experiment, described in appendix
Image-to-caption generation In figure 14 showcase conditional image-to-capele results for the compared models, showing the MMVAE+ is the only approach achieving high quality for the produced sentences, which also fairly coherent in attributes such the colors present in the bird.
Additional comparisons Section 4.2 shows the central result for the comparison between MM- VAE+ and the recently proposed DMVAE (Lee Pavlovic 2021), which also employs factorized latent spaces, The fact that even in relatively simple datasets such as PolyMNIST, alternative models with factorized representations heavily suffer from the shorteut problem, already clearly separates our work from previous approaches. However, for completeness we also test this model in the CUB Image-Captions experiment, ensuring compatibility with MMVAE+ architectures, hyperparam- eters, and other implementation details. We show qualitative and quantitative results for DMVAE on the CUB Image-Caption experiment in figure [17 and table respectively. qualitative and quantitative comparison of results clearly shows MMVAE+ achieves better results than DMVAE in this experiment.
Conditional coherence
Conditional FID
DMVAE
0.487 (10.022))
179. 23 (1.6.99) 164.94 (1:1.50)
MMVAE+ (ours)
0.721 (110.090)
Table 4: Quantitative performance of DMVAE the CUB Image-Caption experiment, compared with MMVAE+, in terms conditional coherence (higher better) and conditional FID (lower is better).
(a) MVAE
(b) MVTCAE
(c) mmJSD
(d) MoPoE-VAE
(c) MMVAE
(f) MMVAE+
Figure 13: Qualitative results for MVAE, MVTCAE, mmJSD MoPoE- VAE, MMVAE and MM- VAE+ for unconditional generation the CUB Image-Captions dataset.
23

---

Published as a conference paper at ICLR 2023
(a) MVAE
(b) MVTCAR
(c) mmJSD
(d) MoPoF-VAR
(c) MMVAR
(f) MMVAF+
Figure 14: Qualitative results for MVAE, MVTCAF, mmJSD, MoPoE- VAE, MMVAE and MM- VAE+ for image-to-caption generation the CUB Image-Captions dataset.
MVAE
(b) MVTCAE
(c) mmJSD
(d) MoPoE-VAE
(c) MMVAE
(f) MMVAE+
Figure 15: Qualitative results for MVAE, MVTСAE, mmJSD MoPoE- VAE, MMVAE and MM- VAE+ for caption-to-image generation the CUB maage-Captions dataset.
24

---

Published as a conference paper at ICLR 2023
MVAE
bb) MVTCAE
(c) MmJJD
(d) MoPoE-VAE
(c) MMVAE
(f) MMVAE+
Figure 16: Qualitative results for the compared models for the caption-to to-imagtt----eeeeeee generation proce- dure used for estimation of generative coherence for the CUB Image-Caption experiment as described in appendix E.2
(a) caption-to-image generation
(b) image-to-caption generation
(c) unconditional generation
Figure 17: Qualitative results for DMVAE the CUB Image-Caption experiment.
25

---

Published as a conference paper at ICLR 2023
H PERFORMANCE OF MMVAE+ AS A FUNCTION OF THE NUMBER OF MODALITIES USED FOR TRAINING
An important trait of successful multimodal learning is that performance should improve, least non-decrease, when additional modalities are available. fact, adding modalities increases, least does not decrease, the information available for the model. However, Daunhawer et al. (2022) highlight that for the prominent approaches in the class of multimodal VAEs having addi- tional modalities causes a decrease generative quality. Here investigate the impact of varying the number of modalities present for training on the MMVAE+ performance. In particular we use the PolyMNIST dataset for an ablation conditional generative coherence and conditional generative quality for MM when varying the number of modalities for training. Note the PolyMNIST dataset consists of five modalities: (mo; mi M2, m3, ma). this ablation, we compare MMVAB+ trained on
five (all) modalities {mo. mi, m2 my m
four modalities {m,, mg, ma, m,}
three modalities {m2, my, ma}
two modalities (ma, m4}
Then, for transparent comparison, we compute FID scores and generative coherences calculated for conditional generation for starting modality m, and target modality m4, and vice-versa, averaging the results. This way we have fair and unbiased comparison of cross wross-generation performance by only considering the last two modalities even models trained with three/four/five modalities.
1.0
150
Conditional coherence
0.8
Conditional FID
100-
0.6
0.4
50+
0.2
0.0
5
2
5
Number of modalities
Number of modalities
Figure 18: MMVAE+ performance for conditional generation on the PolyMNIST dataset, when varying the number of modalities used for training. 3 = 2.5 for all models.
Results suggest that we do not observe a decrease in generative quality when increasing the num- ber of modalities, as is observed for the main existing models in the class of multimodal VAEs (Daunhawer al. 2022). On the other hand, generative coherence improves markedly with more modalities present for training. This result is important and promising, as it suggests that MMVAE+ leverage multiple modalities learn shared information effectively, while the same time not incur in trade-off with generative quality. In other words, having additional modalities seem to only bring benefit, as it is desirable for multimodal VAEs, but currently not achieved for state-of fate-of-the-ane the-art approaches (Daunhawer et al. 2022).
I COMPARING MODELS WITH SEPARATE LATENT SUBSPACES USING DROPOUT DURING TRAINING
In this Appendix, we want to compare the MMVAE+ with existing models with factorized subspaces (see section 4.2). when dropout is applied to such models during training. In fact, Wang et al. (2016) suggest dropout potential solution for the shorteut problem. To this end add dropout when training MMVAE(factorizee)) (ffatt((arrizzee)) and DMVAE on PolyMNIST with the same set- tings in Section 4.2 with equal shared and private latent capacity (setting that corresponds to
26

---

Published as a conference paper at ICLR 2023
100% in the x-axis, in plots in Figure 5) Note that, with this choice for the number of dimensions
for shared and modality- -specific subspaces, all three models trained without dropout clearly show
the presence of shortcut (see again Figure 5 We use different dropout rates: 0.1. 0.2, 0.4; the same values proposed in Wang et al. (016))
To the comparison, we also add VCCA-privaae,, the model proposed by the authors in Wang et al. (2066). Note that for this comparison, extend the VCCA-private model to M modalities and optimize the objective
M
1
pel X W)
LVocA-private (TI:M)
Σ
E Ey (2) [=] Now (WIX)
log
M mmml
admm (z x 19ow (W
where W = W1,- WM. pe 2, W) = p(z) II Pa. (2 Wm)p(Wm))
and
M 40w [W])) m=1 Jo (wm mm
For the compared models, we report in Figure 19 the performance for generative coherence for conditional generation, latent classification accuracy for private subspaces, and generative quality for conditional generation. Dropout rate 0.0 equals to models trained without dropout. In each plot, keep as reference the performance of MMVAE+ trained without dropout, and with the same values for shared and private dimensionalities as all other models. Results show that increasing the dropout rate for existing approaches has a similar effect to constraining the size of private subspaces.
1.0
1.0
130
Slundl conntrt poc.cccccooooi
0.8
0.8
Generalive quality
0.5
0.6
100
04-
20492 (MAA)
sibuchy
0.4
0.2
0.1
0.0
0.4
0.2
0.1
0.0
0.4
0.2
0.1
0.0
Dropout
Dropou
Dropout
(a) Generative coherence (higher is better)
(b) Latent classification accuracy from modality-peccffic encodings
(c) Generative quality (in terms of FID score, lower is better)
(lower is better)
Figure 19: Comparison of MMVAE+, MMVAE((actorizze)) mmJSD((actoriiee)) DMVAE, VCCA- private, trained on PolyMNIST with equal number of dimensions assigned to shared and modality- specific subspaces: both the shared and the modality subspaces are always fixed to 32 di- mensions. (MVAFE(factorizee)) (ffstt(facorrized) DMVAE, VCCA-private are trained with dif- ferent dropout rates, while MM trained without dropout is kept as reference. Note we show dropout rates in decreasing order in the x-axis, to highlight the similarity in trend with the plots from Section 4.2) this shows increasing the dropout rate has similar effect constraining the size of private subspaces.
In particular increasing the dropout rate corresponds to improved generative coherence, and better results for latent classification accuracy for private subspaces. However, this comes at the expense of markedly worsened generative quality. Therefore, increasing the dropout rate seems to prevent the degenerate solution in which all information flows in the modality-specific subspaces. However, also results in low generative quality, which means it is not an effective solution to the shortcut problem, when looking at overall performance. This demonstrates that novel solution of using auxiliary distributions is a more effective method to solve the shortcut problem. From these re- sults it is evident that, even if compared with existing models resorting to dropout during training, MMVAR+ achieves better results when jointly looking at both generative quality and generative coherence.
27


Reviews:
[{"review_id": "nsR3wpQHTVD", "reviewer": "Reviewer_VT76", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a new multi-modal VAE model which includes modality-specific latent variables, as well as shared latent variables with the latter encoded through a mixture-of-experts model.  The new variational bound allows to utilize private latent variables that do not hinder cross-modal coherence and generation, in contrast to previous work where this does not hold in a robust way. Empirical results against different multi-modal VAEs for different benchmark datasets illustrate that the approach improves generative coherence and generative quality.\n\n----\nPost-rebuttal comments:\n\nFollowing the response from the authors and the other reviews, I keep my accept recommendation. \nThe authors have addressed my question as to how the (conditional) generative performance changes when additional modalities are available. The new experiments indicate that generative performance does not decrease, in contrast to some previous work.\nThe additional experiments to address comments from the other reviewers also improve the experimental validation of the suggested approach.\n\n----", "summary_of_the_review": "The paper suggests a novel variational bound for multi-modal VAEs that seems well motivated and appears to yield good empirical results. ", "strength_and_weaknesses": "Strengths:\n- Previous work on multi-modal generative models have struggled to yield both generative coherence and generative quality. The suggested approach illustrates an empirical improvement for both criteria on challenging multimodal datasets. \n- The suggested variational objective is interesting and new, as far as I am aware. It offers a different approach to include modality specific latent variables compared to previous work that appears more robust.\n\nWeaknesses:\n- It is not clear to me if the gap between the variational bound and the log-likelihood does increase with increasing modalities (as in Daunhawer et al., 2022)? \n- Do additional modalities compromise the generative performance? It would be helpful to see how the generative/cross-coherence performance varies empirically, for instance in the PolyMNIST experiments with different numbers of modalities.\n\nActionable feedback:\n- The proportionality following eq. (4) is not clear to me.\n\n\nComments:\n- It appears that the auxiliary prior distributions could be quite general. Is there a specific reason why they have been chosen to be zero centered Gaussians? Does the tightness of the bound relative to the log-likelihood depend on the KL divergence between $p$ and $r$ or $q(w_m|x_m)$ and $r(w_m)$, and could I improve generative quality, for example, by using implicit densities for r(as I do not need to evaluate their density)?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YNVgd1u8ON", "reviewer": "Reviewer_GWdS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work presented in this paper addresses the important problem of multimodal representation learning. Following a large body of work in this domain, the proposed approach builds on an extension of a Variational Auto Encoder (VAE) such that it can encode and decode multiple modalities. The crux of the idea presented in this paper is to partition the latent space into modality-specific and shared latent variables, making sure that the modality-specific latent are not “polluted” by the share information. This is achieved through a modification of the ELBO that originates from the work of Shi et al. ‘19, that uses a mixture of experts approach to obtain the joint approximate posterior. By decomposing the ELBO following Shi et al. ‘19, they observe that for a given modality, the portion that pertains to all other modalities should be trained by assuming a learned “pseudo” prior for their latent variables.\nExperimental results use mainly two datasets: PolyMNIST, which is a variant of MNIST including background images to simulate multiple modalities, and the CUB dataset, which contains two modalities: images and captions. In their experiments, the authors rely on standard metrics from the literature (or their reinterpretation) including generative coherence (both conditional and unconditional) and generative quality (measured through the lenses of the FID score). Experiments serve the purpose of illustrating that the proposed method strikes a good balance between the two objectives (coherence and quality), which is a pain point of previous work.", "summary_of_the_review": "In summary, this work is well presented, well written and the motivation is clear. I think this line of work presents the important trade-off between coherence and generation quality that affects VAE-based method for multimodal representation learning, and this is commendable.\nI have spotted some problems that can be summarized as: 1) given the key idea in this work, it would be possible to conceive a naive baseline to make sure that the claims presented in the paper are indeed correct, and this is missing, 2) Experiments seem not to be conclusive about the superiority of the proposed method with respect to state of the art. There cases in which this seems true, and others in which the “optimality” of the proposed solution cannot be verified.\n\nLastly, since this work is not completely novel, and the strength are not on the methodological contributions, I would advice the authors to improve the experimental section, by considering at least one additional dataset [2], or [3].\n\n[2] Vasco M, Yin H, Melo FS, Paiva A. Leveraging hierarchy in multimodal generative models for effective cross-modality inference. Neural Netw. 2022 Feb;146:238-255. doi: 10.1016/j.neunet.2021.11.019. Epub 2021 Nov 24. PMID: 34906760.\n\n\n[3] @misc{https://doi.org/10.48550/arxiv.2107.07502,\n  url = {https://arxiv.org/abs/2107.07502},\n  author = {Liang, Paul Pu and Lyu, Yiwei and Fan, Xiang and Wu, Zetian and Cheng, Yun and Wu, Jason and Chen, Leslie and Wu, Peter and Lee, Michelle A. and Zhu, Yuke and Salakhutdinov, Ruslan and Morency, Louis-Philippe},\n  title = {MultiBench: Multiscale Benchmarks for Multimodal Representation Learning},\n  publisher = {arXiv},\n  year = {2021},\n}\n\n\n============================\n\nPost rebuttal comments\n\n============================\n\nI've read the various discussions in the rebuttal, and find the new results compelling in verifying the merits of the proposed method. I will update my score from 5 to 8.", "strength_and_weaknesses": "Strength:\n\n* This work follows a line of research that tackle an important problem, that of multimodal representation learning\n\n* The paper is well written, clear, and easy to follow, and builds on many previous work, with the added value of clarifying some aspects of prior work that where not completely clear in the original papers\n\n* Experiments are well conceived \n\nWeaknesses:\n\n* The idea presented in this work (to the best of my understanding) can be summarized as follows: shared information should not pollute modality-specific latent information. To ensure this, a variant of the mixture of expert ELBO is derived, by introducing learnable pseudo-priors that induce the reconstruction of a given modality to rely solely on its modality-specific latent space. A natural question is as follows: since you assume independence between shared and modality-specific latent spaces, and further you constrain modality-specific information to be solely responsible for reconstruction, then why not training individual VAEs, one per modality, then a multimodal VAE to learn a shared representation, and then “stitch” together all latent spaces?\nIn other words, what I miss is a naive baseline to double check that the idea presented in this paper is a valid approach when compared to a simpler method.\n\n* I find Expression (5) in the main paper not very clear. In a previous workshop paper that presents the very same idea and the very same method I find the use of a tilde on the “other-modalities” private latent variables easier to understand and more clear. On the good side, I think the textual explanations of the ELBO are much more useful in this version of the paper.\n\n* In the experiments, unconditional coherence seem to be a more difficult objective to attain, as was also shown in prior works. In this case, we usually sample from the prior on the latent space, and only rely on decoders to generate each modality. Then we check the coherence of each generated modality. 1) It would be useful to clarify how does this work when you have a latent space that is segregated. I assume you take the prior of each modality-specific latent, and the shared prior, but it would be good to state this clearly (in the appendix). Why do you only have results on PolyMNIST for this challenging metric, and not for CUB? \nMoreover, when we look at table 3 in the appendix (for PolyMNIST), we can see that the proposed method really struggles to achieve high generation quality (there, the MVAE approach based on PoE works better). It is also interesting to note that additional “tricks” to improve base models (importance sampling, double reparametrization), seem to hurt generation quality, which is not expected.\n\n* In the experiments, when we look at “competitors”, we learn that if the capacity is sufficient, then the modality-specific encoder can learn all the information (both shared and private) such that an adverse effect of a shortcut appears. But if all information is stored in the modality-specific latent space, and this is true for each modality, then wouldn’t this be enough to achieve coherence? Aren’t we simply looking at the “fallacy” of the MoE approach that, by construction, samples only one of the components for the generation?\n\n* Fig 5 raise several questions. \n** Fig 5b indicates latent classification accuracy, obtained by training a simple linear model on “freezed” encoders that produce latent features. It seems counter intuitive to state that lower accuracy is better. This is because the goal is to show that if we only use modality-specific encodings we should not be able to classify correctly, if the hypothesis of no shared information being present in those latent spaces. Beside the fact that I am not convinced that this metric (first introduced in Shi et al ‘19) is useful to state anything about the latent space natural segregation, I think that comments to this figure should be more informative. \n** Fig 5 c: I assume this figure is for the conditional generation right? Since there are works that criticize the use of FID score to assess the generative quality [1], I wonder why we do not have a table with likelihood values, or Bit per Dimension, which can also be useful to assess the quality of the model.\n** Overall, from the three sub-figures in fig 5, we can remark that mmJSD is a close competitor to MMVAE+. I think this should be acknowledged and commented.\n\n* The last paragraph before section 4.3 claims for an optimal balance between coherence and generation quality. How can you claim optimality?\n\n* In section 4.3, qualitative results for the CUB dataset, in terms of coherence, use an original metric discussed in the appendix. This is an important detail that I think should be discussed at least a minimum in the main paper. Indeed, the proposed metric is somehow questionable and should be put upfront for the reader to properly understand. In my opinion, it is not clear why you could have not used the same technique used in Shi et al ‘19, by taking the learned representation of your ResNet, in a similar way as if you used a pre-trained network. The introduction of artificial captions and the count of correctly colored pixels suffers from the problems stated by the authors, e.g. counting only the background color. Using the top-2 most frequent pixel colors might work in some cases an not others, so I am not sure how much can we rely on this coherence metric. Also, as commented above, this is conditional coherence, which seems an easier task to achieve than unconditional coherence. Do you have any comments why, and why didn’t you show unconditional coherence metric fo CUB?\n\n\n[1] @misc{FID_critics,\n  url = {https://arxiv.org/abs/2203.06026},\n  author = {Kynkäänniemi, Tuomas and Karras, Tero and Aittala, Miika and Aila, Timo and Lehtinen, Jaakko},\n  title = {The Role of ImageNet Classes in Fréchet Inception Distance},\n  publisher = {arXiv},\n  year = {2022},\n}\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QZm2mUUhTdL", "reviewer": "Reviewer_i3y6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a modified version of the existing MMVAE model called MMVAE+ for weakly-supervised generative learning with multiple modalities. The paper aims to overcome the trade-off between generative quality and generative coherence by suggesting having separate latent encoding for both shared and private features within each modality and a new ELBO. The authors also introduce auxiliary distributions for private features to achieve the cross-modal reconstruction. Experiments show their model achieves both good generative coherence and high generative quality in challenging experiments.\n", "summary_of_the_review": "Although the paper is well written and the arguments are well justified, the paper lacks technical and empirical novelty to contribute as the problem and solution has been raised by an existing publication. \n\n-----------------------------------------------------------------------\nPost rebuttal comments:\n\nAfter reading the rebuttal and the other reviews, we raise the score to 6.\n\nAs one of the contributions is the auxiliary distribution, it would be interesting to see how different choices of auxiliary distributions would affect the experiment results.", "strength_and_weaknesses": "Strengths：\nThe paper is well-written, clear and well-organized.\nExperiments show their model achieves both good generative coherence and high generative quality and is robust to the size of the private space.\n\nWeaknesses:\n(Lee, Pavlovic 2021) should also be included as a baseline. \nIt would be good to have an ablation study for the auxiliary distributions.\nThe figures can be improved:\nIt would be better to have an explanation on the boxes, lines and dashed lines for Figure 2.\nIt would be better to have an explanation for the other rows in Figure 3, why do they come in blocks?\nThe axis should be consistent in Figure 5. In (b),  lower is better but 0 is at the top, which confuses the readers.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster