Learning Multimodal VAEs through Mutual Supervision
Authors: Tom Joy
Yuge Shi
Philip Torr
Tom Rainforth
Sebastian M Schmon
Siddharth N
Venue: ICLR 2022 Spotlight (2022)
Keywords: Multimodal Variational Autoencoder
Variational Autoencoder

Abstract: Multimodal VAEs seek to model the joint distribution over heterogeneous data (e.g.\ vision, language), whilst also capturing a shared representation across such modalities. Prior work has typically combined information from the modalities by reconciling idiosyncratic representations directly in the recognition model through explicit products, mixtures, or other such factorisations. Here we introduce a novel alternative, the MEME, that avoids such explicit combinations by repurposing semi-supervised VAEs to combine information between modalities implicitly through mutual supervision. This formulation naturally allows learning from partially-observed data where some modalities can be entirely missing---something that most existing approaches either cannot handle, or do so to a limited extent. We demonstrate that MEME outperforms baselines on standard metrics across both partial and complete observation schemes on the MNIST-SVHN (image--image) and CUB (image--text) datasets. We also contrast the quality of the representations learnt by mutual supervision against standard approaches and observe interesting trends in its ability to capture relatedness between data.

Published as a conference paper at ICLR 2022
LEARNING MULTIMODAL VAES THROUGH MUTUAL SUPERVISION
Tom Joy', Yuge Shi', Philip H.S. Torr', Tom Rainforth' Sebastian Schmon', N. Siddharth
University of Oxford
University of Durham
University Edinburgh & The Alan Turing Institute
tomjeyerobots.. ox.ac.uk
ABSTRACT
Multimodal variational autoencoders (VAEs) seek to model the joint distribution over heterogeneous data (e.g. vision, language), whilst also capturing shared representation across such modalities. Prior work has typically combined infor- mation from the modalities by reconciling idiosyneratic representations directly in the recognition model through explicit products, mixtures, or other such factorisa- tions. Here we introduce novel alternative, the Mutually supErvised Multimodal VAE (MEME), that avoids such explicit combinations by repurposing semi- supervised VAEs to combine information between modalities implicitly through mutual supervision. This formulation naturally allows learning from partially- observed data where some modalities can be entirely missing-something something that most existing approaches either cannot handle, or do so to limited extent. We demonstrate that MEME outperforms baselines on standard metrics across both partial and complete observation schemes on the MNIST-SVHN (image-image) and CUB (image-text) dataset: We also contrast the quality of the representa- tions learnt by mutual supervision against standard approaches and observe inter- esting trends its ability to capture relatedness between data.
INTRODUCTION
Modelling the generative process underlying heterogenous data, particularly data spanning multiple perceptual modalities such as vision language, can be enormously challenging. Consider for ex- ample, the case where data spans across photographs and sketches of objects. Here, a data point, comprising of an instance from each modality, constrained by the fact that the instances are re- lated and must depict the same underlying abstract concept. An effective model not only needs to faithfully generate data in each of the different modalities, it also needs do in manner that preserves the underlying relation between modalities. Learning a model over multimodal data thus relies on the ability bring together information from idiosyncratic sources in such a way as to overlap aspects they relate on, while remaining disjoint otherwise.
Variational autoencoders (VAEs) (Kingma & Welling 2014) are a class of deep generative mod- els that are particularly well-suited for multimodal data as they employ the use of encoders- learnable mappings from high-dimensional data to lower-dimensional representations- provide the means combine information across modalities. They also be adapted to work in situations where instances are missing for some modalities; a common problem where there are inherent diffi- culties in obtaining and curating heterogenous data. Much of the work in multimodal VAEs involves exploring different ways to model and formalise the combination of information with a view to im- proving the quality of the learnt models (see 2
Prior approaches typically combine information through explicit specification as products (Wu &
Goodman 2018), mixtures (Shi al. 2019), combinations of such (Sutter et al. (021). or through additional regularisers on the representations (Suzuki et al.) 2016 Sutter et al. 2020). Here, we
explore an alternative approach that leverages advances in semi sumi-supervised VAEs Siddharth al.]
'The codebase is available at the following location: http::/gtthaacccom//wyyymmmee

---

Published as a conference paper at ICLR 2022
Data
Modaily1
Modality 2
Modailly
Modsllty 2
Modaliy
Modailty
Rio
Aog
Encoding
Prtor
Ensoding
Encoding
Prior
Encocing
Encoding (prior)
Encoding (prior)
Encoding
(a) VAE
(b) Typical Multimodal VAE
(c) MEME (ours)
Figure 1: Constraints on the representations. (a) VAE: A regularsss the data encoding distrihution through KL. (b) Typical multimodal VAE: Encodings for different modalities are first explicitly combined, with the result regularised by prior through KL. MEME (ours): Leverage semi-supervised VAEs cast modality as conditional prior, implicitly supervisingrreguuariiinng the other through VAE's KL. Mirroring the arrangement to account for KL asymmetry enables multimodal VAEs through mutual supervision.
2017
repurpose existing regularisation in the VAE framework as an implicit means by which information is combined across modalities Figure T).
Joy al. 2021
We develop novel formulation for multimodal VAËs that views the combination of information through a semi-supervised lens, as mutuall supervision between modalities. We term this approach Mutually supErvised Multimodal VAE (MEME). Our approach not only avoids the need for ad- ditional explicit combinations, but it also naturally extends to learning in the partially-obsenve setting- something that most prior approaches cannot handle. We evaluate MEME on standard metrics for multimodal VA Es across both partial and complete data settings, on the typical multi- modal data domains, MNIST-SVI CUB (image-text), and show that it outperforms prior work on both. We additionally investigate the capability of MEMEs ability to capture the 'relatedness' a notion of semantic similarity, between modalities in the latent representation; in this setting we also find that MEME outperforms prior work considerably.
(image-image) and the less common but notably more complex
RELATED WORK
Prior approaches to multimodal VAEs can be broadly categorised in terms of the explicit combina- tion of representations (distributionss, namely concatenation and factorization.
Concatenation: Models in this category learn joint representation by either concatenating the inputs themselves or their modality-specific representations. Examples for the former includes early work multimodal VAEs such as the JMVAE (Suzuki et al. (016)). triple ELBO (Vedantam et al. 2018) and MFM (Tsi et al. 2019). which define joint encoder over concatenated multimodal data. Such approaches usually require the training of auxiliary modality-specific components handle the partially-observed setting, with missing modalities, at test time. They also cannot learn from partially-observed data. In very recent work. Gong et al. (2021) propose VSAE where the latent representation is constructed as the concatenation of modalityspeccifi encoders. Inspired by VA that deal with imputing pixels in images such as VAEAC (Ivanov et al., 2019), Partial VAE (Ma) et al., 2018), MIWAE (Mattei & Frellsen 2019). HI-VAE (Nazábal et al. 2020) and pattern-set setting by incorporating modality mask. This, however, introduces additional components such as collective proposal network and a mask generative network, while ignoring the need for the joint distribution over data to capture some notion of the relatedness between modalities.
mixture model (Ghalebikesabi et al. (021). VSAE can learn in the
Factorization: In order to handle missing data at test time without auxiliary components, recent work propose factorize the posterior over all modalities the product (Wu & Goodman 2018) or mixture (Shi et al., 2019) of modality-specifff posteriors (experts). Following this, /utter et al. (2021) proposes to combine the two approaches (MoPoE-VALE to improve learning in settings where the number of modalities exceeds two. In contrast to these methods, mmJSD (Sutter et al., 2020) combines information not the posterior, but "dynamic prior", defined as function (either mixture product) over the modality posteriors well pre-defined prior.
Table I provides a high-level summary of prior work. Note that all the prior approaches have some explicit form of joint representation distribution, where some of them induces the need for auxiliary components to deal with missing data at test time, while others are established without significant theoretical benefits. By building upon semi-supervised framework, our method MEME circumvents this issue to learn representations through mutual supervision between modalities, and is able to deal with missing data at train or test time naturally without additional components.
2

---

Published as a conference paper at ICLR 2022
Table I: We examine four characteristics: The ability to handle partial observation at test and train time, the form of the joint distribution or representation in the bi-modal case (s. t are modalities), and additional components. (V) indicates theoretical capability that is not verified empirically.
Partial Test Partial Train
Joint repr./dist.
Additional
JMVAE
(2 t) 9o (z
Io. (..(zz8) (z]s), qo. (2.(2)))
ELBO
XX
go. (z(s), qo. (z(t)
MFM
qu ( t)
ge, (z|s), qo. (z]t)
VSVAE
concat(E,, 2x)
mask generative network
MVAE
ge. (z)s)) (z t)p(z)
sub-sampling
MMVAE
qó, (z|s) + qo, (z]t)
MoPoE
go. (z|s) + qo, (zit) q6. (z)s) (z|t)
mMJSD
X
flg. (2]8), qo (z))), p()))
Ours
3 METHOD
Consider a scenario where we are given data spanning two modalities, and curated pairs (s. For example this could be an "image" and associated "caption" of an observed scene. We will further assume that some proportion of observations have one of the modalities missing. leaving us with from both modalities, and Dx De for the proportion containing observations only from modality and respectively, we can decompose the data as D = Dz U D, U.D...
data. Using Dx,: to denote the proportion containing fully observed pairs
In aid of clarity, we will introduce our method by confining attention this bi-modal case, providing discussion on generalising beyond two modalities later. Following established notation in the literature on VAËs, we will denote the generative model using p, latent variable using z, and encoder, recognition model, using q. Subscripts for the generative and recognition models, where indicated, denote the parameters of deep neural networks associated with that model.
3.1 SEMI-SUPERVISED VABS
To develop our approach we draw inspiration from semi-supervised VAEs which use additional information, typically data labels, to extend the generative model. This facilitates learning tasks such as disentangling latent representations and performing intervention through conditional generation. In particular, we will build upon the work of Joy et al. (2021), who suggests to supervise latent representations in VA with partial label information by forcing the encoder, or recognition model, to channel the flow of information as s + z - t. They demonstrate that the model learns latent representations, z, of data, s, that can be faithfully identified with label information
Figure 2) shows modified version of the graphical model from Joy al. (202T), extracting just the salient components, and avoiding additional constraints therein. The label, here t, is denoted as par- tially observed as not all observations have associated labels. Note that, following the information flow argument, the generative model factorises as Pe (s, z, t)) (s z) p (z t)) p(t)) (solid arrows) whereas the recognition model factorises go.p (t, s) (t 2) qs s) (dashed arrows). This autoregressive formulation both the generative and recognition models is what enables the "su- pervision" " of the latent representation s by the label, t, via the conditional prior p.p (z t) as well the classifier (t z).
Py (x
t
(t 2)
Figure 2: Simplified graphical
model from Jompoy et al. (202T)
The corresponding objective for supervised data, derived as the (negative) variational free energy or evidence lower bound (ELBO) of the model is
Y (t|z) (z(s)
log po(siz)pp (z)t)
log (s. t)
Egg (m/s)
+ log q0.p (t|s) (1)
go go( (t/z)
with the generative and recognition model parameterised by θ = {0, 0} and + = (0. respec-
tively. A derivation of this objective can be found in Appendix A
3

---

Published as a conference paper at ICLR 2022
3.2 MUTUAL SUPERVISION
Procedurally, a semi-supervised VAE is already multimodal. Beyond viewing labels as a separate data modality, for more typical multimodal data (vision, language), one would just need to replace labels with data from the appropriate modality, and adjust the corresponding encoder and decoder to handle such data. Conceptually however, this simple replacement can be problematic.
Supervised learning encapsulates very specific imbalance in information between observed data and the labels-that labels do not encode information beyond what available in the observation itself. This is consequence of the fact that labels are typically characterised as projections of the data into some lower-dimensional conceptual subspace such as the set of object classes one may encounter in images, for example. Such projections cannot introduce additional information into the system, implying that the information in the data subsumes the information in the labels, i.e. that the conditional entropy of label t given data is zero: (t s) O. Supervision-base based models typically incorporate this information imbalance as feature, as observed in the specific correspondences and structuring enforced between their label y and latent in Joy et al. (202T).
Multimodal data of the kind considered here, on the other hand, does not exhibit this feature. Rather than being characterised as a projection from one modality to another, they are better understood as idiosyncratic projections of an abstract concept into distinct modalities for example, as an image of a bird or a textual description of it. In this setting, no one modality has all the information, as each modality can encode unique perspectives opaque to the other. More formally, this implies that both the conditional entropies H (t s) and H(s t) are finite.
Based on this insight symmetrise the semi-supervised VAE formulation by additionally con- structing a mirrored version, where we swap s and t along with their corresponding parameters, i.e. the generative model now uses the parameters & and the recognition model now uses the parameters @. This has the effect of also incorporating the information flow in the opposite direction to the stan- dard case as t + - s, ensuring that the modalities are now mutually supervised. This approach forces each encoder to act as an encoding distribution when information flows one way, but also act prior distribution when the information flows the other way. Extending the semi-supervised VAE objective (6). we construct a bi-directional objective for MEME
La t) = (e (s. - (t,s),
where both information flows are weighted equally. On practical note, we find that it is important to ensure that parameters are shared appropriately when mirroring the terms, and that the variance in the gradient estimator is controlled effectively. Please see Appendices to D for further details.
3.3 LEARNING FROM PARTIAL OBSERVATIONS
In practice, prohibitive costs on multimodal data collection and curation imply that observations can frequently be partial, i.e., have missing modalities. One of the main benefits of the method account of its semi-
introduced here is its natural extension to the case of partial observations supervised underpinnings. Consider, without loss of generality, the case where we observe modality but not its pair Recalling the autoregressive generative model p(s, z, t) p(s z)p(z t)p(t) we can derive a lower bound on the log-evidence
log pa.v (s) = log fro(s z)p (z t))(()) dz dt 2 (mx) log po(s 2) fp t))(t) dt
(3)
gz(z
Estimating the integral p(z) = p(z t))(t) dt highlights another conceptual difference between a (semi-)supervisee setting and multimodal one. When t is seen label, this typically implies that one could possibly compute the integral exactly by explicit marginalisation over its support, or at the very least, construct a reasonable estimate through simple Monte-Carll integration. In Joy ctt al. (202I), the authors extend the latter approach through importance sampling with the "inner" encoder q( z), to construct looser lower bound to (3).
In the multimodal setting however, this poses serious difficulties as the domain of the variable t is not simple categorical labels, but rather complex continuous-valud data. This rules out exact marginalisation, and renders further quality of samples expect from the encoder q(t z) which itself is being learnt from
sampling practically infeasible on account of the
4

---

Published as a conference paper at ICLR 2022
data. To overcome this issue and to ensure flexible alternative, adopt approach inspired by the VampPrior (Tomczak & Welling 2018). Noting that our formulation includes prior pu(z t), we introduce learnable
a conditional X {ut to estimate the prior as Px.(z) - E Polz uf). Our objective for when t is unobserved is thus
log Po 2))x+(2)
(s 2)) g s)
N + log Σ pola uf)
(ss) (xls))
(xls))
log
(4)
gø(z
where the equivalent objective for when s is missing can be derived in a similar way. For a dataset D containing partial observations the overall objective (to maximise) becomes
log (s,
Σ C(s) NED,
E(()) +
Σ C(t) LED
+
Σ S,tEDs
CBi(s,t),
(5)
X.ED
This treatment of unobserved data distinguishes our approach from alternatives such as that of Shi et al. (2019), where model updates for missing modalities are infeasible. Whilst there is the possibil- ity to perform multimodal learning in the weakly supervised case as introduced by Wu & Goodman (2018), their approach directly affects the posterior distribution, whereas ours only affects the reg- ularization of the embedding during training. At test time, WW & Goodman (018) will produce different embeddings depending on whether all modalities are present, which is typically at odds with the concept of placing the embeddings of related modalities in the same region of the latent space. Our approach does not suffer from this issue as the posterior remains unchanged regardless of whether the other modality is present or not.
Learning with MEME Given the overall objective in (5), train MEME through maximum- likelihood estimation of the objective over a dataset D. Each observation from the dataset is op- timised using the relevant term in the right-hand side of ()) through the use of standard stocastic gradient descent methods. Note that training the objective involves learning all the (neural network) parameters (0, vb o, (p) in the obsly-observed, bi-directional case. When training with a partial ob- servation, say just s, all parameters except the relevant likelihood parameter yo (for (t 2)) are learnt. Note that the encoding for data in the domain of t is still computed through the learnable pseudo-samples Xt. This is reversed when training on an observation with just t.
Generalisation beyond two modalities We confine attention here to the bi-modal case for two important reasons. Firstly, the number of modalities one typically encounters in the multimodal setting is fairly small to begin with. This is often consequence of its motivation from embodied perception, where one is restricted by the relatively small number of senses available (e.g. sight, sound, proprioception). Furthermore, the vast majority of prior work on multimodal VAEs only really consider the bimodal setting (cf. |S 2}. Secondly, it is quite straightforward to extend MEMÉ to settings beyond the bimodal case, by simply incorporating existing explicit combinations (e.g. mixtures or products) on top of the implicit combination discussed here, we provide further expla- nation AAppendx Our focus in this work lies in exploring and analysing the utility of implicit combination in the multimodal setting, and our formulation and experiments reflect this focus.
EXPERIMENTS
4.1 LEARNING FROM PARTIALLY OBSERVED DATA
In this section, we evaluate the performance of MEME following standard multimodal VAE metries as proposed in Shi et al. (201)). Since our model benefits from its implicit latent regularisation and able to learn from partially-observe proportions of data are missing in either or both modalities during training. The two metrics used are cross coherence to evaluate the semantic consistency in the reconstructions, as well as latent accuracy in a classification task to quantitatively evaluate the representation learnt in the latent space. We demonstrate our results on two datasets, namely an image < image dataset MNIST- SVHN (LeCun et al. 2010 Netzer al. 2011), which is commonly used to evaluate multimodal VAEs (Shi et al. 2019 Shi et al.) [021]] /Stter et al.) 2020 [20I];: as well as the more challenging, but less common, image <+ caption dataset CUB WWelinder et al. 2010).
data, here we evaluate MEME's performance when different
5

---

Published as a conference paper at ICLR 2022
1
2
3
4
5
6
7
8
00
15
3
4
5
66n
7
Input
5
8
0
2
3
5
6
7
9
Output
Figure MEME cross-modal generations for MNIST-SVHN.
being this bird has a bird brown and and and very short beak.
this is a bird with red breast and red head.
this bird has a black top and yellow battom with black lines the head and beak are small.
distinct this bird has wings that are black and has orange belly.
most this bird has wings that are green and has an red belly
this is a large black bird with long neck and bright orange cheek patches.
Figure 4: MEME cross modal generations for CUB.
Following standard approaches, we represented image likelihoods using Laplace distributions, and a categorical distribution for caption data. The latent variables are parameterised by Gaussian distri- butions. In line with previous research (Shi et al. 2019, Massiceti et al. 2018), simple convolutional architectures were used for both MNIST-SVHN and for CUB images and captions. For details on training and exact architectures see Appendix K. we also provide tabularised results in Anppeddix H.
Cross Coherence Here, we focus mainly on the model's ability to reconstruct one modality, say, i, given another modality, s, input, while preserving the conceptual commonality between the two. In keeping with Shi et al. (2019), we report the cross coherence score on MNIST-SVHN as the percentage of matching digit predictions of the input and output modality obtained from a pre- trainee classifier. On CUB we perform canonical correlation analysis (CCA) on input-output pairs of cross generation to measure the correlation between these samples. For more details on the computation of CCA values we refer to Apppediii G
In Figure 5) we plot cross coherence for MNIST-SVHN and display correlation results for CUB in Figure 6] across different partial-observation schemes. The x-axis represents the proportion of data that is paired, while the subscript to the method (see legends) indicates the modality that presented. with ʃ = 0.25 indicates that only 25% of samples are paired, and the other 75% only contain MNIST digits, and MEME.SPLIT with f = 0.25 indicates that the 75% contains mix of MNIST and SVHN samples that are unpaired and never observed together, i.c we alternate depending on the iteration, the remaining 25% contain paired samples. We provide qualitative results in Figure and FFigure
For instance,
We can see that our model is able to obtain higher coherence scores than the baselines including MVAE (Wu & Goodman 20188) and MMVAE (Shi et al.) (201)) in the fully observed case, / 1.0, as well as in the case of partial observations, f 1.0. This holds true for both MNIST-SVHN and C It is worth pointing out that the coherence between SVHN and MNIST is similar for both partially observing MNIST or SVHN. i.c. generating MNIST digits from SVHN is more robust to which modalities are observed during training (Figure Right). However, when generating SVHN from MNIST, this is not the case, when partially observing MNIST during training the model struggles to generate appropriate SVHN digits. This behaviour somewhat expected since the information needed to generate MNIST digit is typically subsumed within an SVHN digit (e.g. there is little style information associated with MNIST), making generation from SVHN to MNIST easier, and from MNIST to SVHN more difficult. Moreover, MNIST during training provides greater clustering in the latent space, which seems to aid cross generating SVHN digits. We provide additional t-SNE plots in Appendix H.3 to justify this claim.
also hypothesise that observing
For CUB we can see in Figure 6 that MEME consistently obtains higher correlations than MVAE across all supervision rates, and higher than MMVAE in the fully supervised case. Generally, cross- generating images yields higher correlation values, possibly due to the difficulty in generating se- mantically meaningful text with relatively simplistic convolutional architectures. We would like to highlight that partially observing captions typically leads to poorer performance when cross-
We note that some of the reported results of MMVAF in our experiments do not match those seen in the original paper, please visit Appendix for more information.

---

Published as a conference paper at ICLR 2022
Coherence Between NIUST and SVHN
Coherence Between SVHN and HNIST
07-
05-
VENVE JAMST
HERE JOEST
VENT SUIN
HERE_5SM
04
SENL
03-
VNAE MRAST
MAE HUST
VIAA SWIN
MAAE SWIN
03-
VNAVE SRLTT
MAK SP.I
02-
0.2925
2.225
0.29
0.0025
0.125
0.23
1.0
Stsarvation frattion
Dteai valion facton
Figure 5: Coherence between MNIST and SVHN (Left) and SVHN and MNIST (Right). Shaded area indicates one-standard deviation of runs with different seeds.
Comelation Betwaon Image Gerorated Sentence
Corretation Bctwoon Santenoe Generated Image
0.80
0.20
0.25
0.20
0.30
Capton
moze
0.1
mago
0.50
0.09
0.00
-0.05
125
23
03
0.13
Figure 6: Correlation between Image and Sentence (Left) and Sentence and Image (Right). Shaded area indi- cates one-standard deviation of runs with different seeds.
generating captions. We hypothesise that is due to the difficulty in generating the captions and the fact there is a limited amount of captions data in this setting.
To gauge the quality of the learnt representations we follow previous work (Hig- gins et al. 2017 Kim & Mnih 2018 Shi et al., 2019 Sutter et al. (021) and fit a linear classifier that predicts the input digit from the latent samples. The accuracy of predicting the input digit using this classifier indicates how well the latents can be separated in a linear manner.
Latent Accuracy
In Figure 7] we plot the latent accuracy on MNIST and SVHN against the fraction of observation. We can see that MEME outperforms MVAE on both MNIST and SVHN under the fully-observed scheme (i.e. when observation fractions is 1.0). We can also notice that the latent accuracy of MVAE is rather lopsided, with the performance on MNIST to be as high as 0.88 when only 1/16 of the data is observed, while SVHN predictions remain almost random even when all data are used; this indicates that MVAE relies heavily on MNIST to extract digit information. On the other hand, MEME's latent accuracy observes steady increase as observation fractions grow in both modali- ties. It is worth noting that both models performs better on MNIST than SVHN in general-this is unsurprising as it is easier to disentangle digit information from MNIST, however our experiments here show that MEME does not completely disregard the digits in SVHN like MVAE does, resulting in more balanced learned representations. It is also interesting to that MVAE obtains higher latent accuracy than MEME for low supervision rates. This is due to MVAE learning to construct representations for each modality in completely separate sub-space in the latent space, we provide a t-SNE plot to demonstrate this in Appendix H.1
Ablation Studies To study the effect of modelling and data choices on performance, we perform two ablation studies: one varying the number of pacudo-samples for the prior, and the other evalu- ating how well the model leverages partially observed data over fully observed data. We find that performance degrades, as expected, with fewer aseudo-sampley and that the model trained with additional partially observed data does indeed improve. See Seel Apeendpxxx ]oor details.
7

---

Published as a conference paper at ICLR 2022
Latent Accuracy MNIST
Latent Accuracy SVHN
57
HLME JIRET
0.90
HOME SPLIT
RUKE MNITT
0.89
0.3
KNGE SA-NN
MAKE_SPLT
HVVAE
0.50
0.54
MENE_PAIST
DVMM
0.3
0.82
FRAST
0.
01
0.78
0.2925
2.125
0.29
0.0025
0.125
Dteai valion faction
Figure 7: Latent accuracies for MNIST and SVHN (Left) and SVHN and MNIST (Right). Shaded area indicates one-standard deviation of runs with different seeds.
4.2 EVALUATING RELATEDNESS
Now that we have established that the representation learned by MEME contains rich class infor- mation from the inputs, also wish analyse the relationship between the encodings of different modalities by studying their "relatedness", i.e. semantic similarity. The probabilistic nature of the learned representations suggests the use of probability distance functions as measure of related- ness, where a low distance implies closely related representations and vice versa.
In the following experiments closed-form expression for Gaussian distributions (see [Appendix Ffor more details). Specifically, we compute de W2( g(z)), || gz (ttttt,))),)r ). where (z((s:) and q(z)tz) are the individual encoders, for all combination of pairs {s., tý } in the mini-batch, i.e {s;, ) for i, M. where M the number of elements in the mini-batch.
the 2- Wassersttin distance, W2, probability metric with
General Relatedness In this experiment we wish to highlight the disparity in measured relat- edness between paired vs. unpaired multimodal data. To do so, we plot day on a histogram and color-code the histogram by whether the corresponding data pair {s;, tú} shows the same concept, e.g. same digit for MNIST-SVHN and same image-caption pair for CUB. Ideally, we should observe smaller distances between encoding distributions for data pairs that related, and larger for ones that are not.
To investigate this, we plot de histogram for every mini-batch ideally should see higher densities at closer distances for points that are paired, and higher densities at further distances for unpaired points. In Figure 8, that MEME (left) does in fact yields higher mass at lower
Hattgram Wasserten distance
Histgramm am of Possentein distance
Hiltogram Wasserttii distance
Natee Nared
Us-Ranee
Punnel
Panod
3530
Mistotion Disawee
Monserter Desance
Wasseston 26000900
Wasserteln
Histgrram
distances
Ln Hare
Uhs Parna
2. 25 100
Figure 8: Histograms of Wassertein distance for SVHN and MNIST (Top) and CUB (Bottom): MEME (Left). MMVAE (middle) and MVAE (Right). Blue indicates unpaired samples and orange paired samples. We expect to see high densities of blue at further distances and visa-versa for orange.
8

---

Published as a conference paper at ICLR 2022
distance values for paired multimodal samples (orange) than it does for unpaired ones (blue). This effect is not so pronounced in MMVAE and not present at all in MVAE. This demonstrates MEME's capability of capturing relatedness between multimodal samples in its latent space, and the quality of its representation.
Class-contextual Relatedness To offer more insights on the relatedness of representations within classes, construct a distance matrix K 10 for the MNIST-SVHN dataset, where each element K corresponds to the average W2 distance between encoding distributions of class i of MNIST and j of SVHN. A perfect distance matrix will consist of a diagonal of all zeros and positive values in the off-diagonal.
See the class distance matrix in Figure 9 (top row), generated with models trained fully observed multimodal data. It is clear that our model (left) produces much lower distances on the diagonal, i.e. when input classes for the modalities are the same, and higher distances off diagonal where input classes are different. A clear, lower-valued diagonal can also be observed for MMVAE (middle), however is less distinct compared MEME, since some of the mismatched pairs also obtains smaller values. The distance matrix for MVAE (right), on the other hand, does not display a diagonal all, reflecting poor ability identify relatedness or extract class information through the latent.
To closely examine which digits are considered similar by the model, we construct dendrograms to visualise the hierarchical clustering digits by relatedness, as seen inlFigure (bottom row). We see that our model (left) is able to obtain clustering of conceptually similar digits. In particular, digits with smoother writing profile such as 3, 5, 8, along with and 9 are clustered together (right hand side of dendrogram), and the digits with sharp angles, such as 4 and are clustered together. The same trend is not observed for MMVAE nor MVAE.
also important to note the height of each bin, where higher values indicate greater distance between clusters. Generally the clusters obtained in MEME are further separated for MMVAE, demonstrating more distinct clustering across classes.
Average Wassertein divergence
Ave veerage Wessercein divergence
Average Wascertein divergence
WUST OISTO
Oig
SAW 15
SWIN Dight
SW-N Si-g
Clustering Dendrogram
Clustering Dendrogram
Clustering Dendrogram
0.00
0.60
0.60
0.45
0.45
0.45
0.30
0.30
0.30
0.15
Jioit
DiaR
Figure 9: Distance matrices for KL divergence between classes for SVHN and MNIST (Top) and dendrogram
(Bottom) for: Ours (Left), MMVAE (middle) and MVAE (Right).
DISCUSSION
Here we have presented a method which faithfully deals with partially observed modalities in VAEs. Through leveraging recent advances semi-supervised VAEs, we construct model which amenable to multi-modal learning when modalities are partially observed. Specifically, our method employs mutual supervision by treating the uni-modal encoders individually and minimizing KL between them to ensure embeddings for are pertinent to one another. This approach enables to successfully learn a model when either of the modalites are partially observed. Furthermore, our model is able to naturally extract indication of relatedness between modalities. We demonstrate our approach on the MNIST-SVIH and CUB datasets, where training is performed on a variety of different observations rates.

---

Published as a conference paper at ICLR 2022
Ethics Statement We believe there inherent ethical concerns within this work, all datasets and motivations do not include or concern humans. As with every technological advance- ment there is always the potential for miss-use. for this work though, where this method may act adversarial to society. In fact, we believe that multi-modal representa- tion learning in general holds many benefits, for instance in language translation which removes the need to translate to base language (normally English) first.
not situation
REFERENCES
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146, 2017.
Yarin Gal. Uncertainty in deep learning. PhD thesis, University of Cambridge, 2016. Unpublished doctoral dissertation.
Sahra Ghalebikesabi, Rob Cornish, Chris Holmes, and Luke J. Kelly. Deep generative missingness pattern-set mixture models. In AISTATS, pp. 3727-3735, 2021. 2021.
Clark R. Givens and Rae Michael Shortt. class of Wasserstein metrics for probability distribu- tions., 2002. ISSN 0026-2285.
Yu Gong, Hossein Hajimirsadeghi, Jiawei He, Thibaut Durand, and Greg Mori. Variational selective autoencoder: Learning from partially-obserye heterogeneous data. In AISTATS, pp. 2377-2385, 2021.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR 2017 International Conference on Learning Repre- sentations 2017. 2017.
Oleg Ivanov, Michael Figurnov, and Dmitry P. Vetrov. Variational autoencoder with arbitrary con- ditioning. International Conference on Learning Representations, pp. 1-25, 2019.
Tom Joy, Sebastian Schmon, Philip Torr, Siddharth N, and Tom Rainforth. Capturing label char- acteristics in {vae)s. In International Conference Learning Representations, 2021. URL
https:://ppeneevieee oor.
net
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pp. 2649-26555 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann. wew.nnannnllenn....m wtwp.e/ww..w./e/////iiiiii exmmmmix//miss, 2, 2010.
Chao Ma. Sebastian Tschiatschek. Konstantina Palla, José Miguel Hernández-Lobato, Sebastian Nowozin, and Cheng Zhang. Eddi: Efficient dynamic discovery of high-value information with partial vae. In International Conference Muchine Leurning, pp. 4234 4243, 2018.
Daniela Massiceti, N. Siddharth, Puneet K. Dokania, and Philip H.S. Torr FlipDial: generative model for two-way visual dialogue. In IEEE Conference on Computer Vision and Pattern Recog- nition, 2018.
Pleare-Alexandre Mattei and Jes Frellsen. Míwae: Deep generative modelling and imputation of incomplete data sets. In International Conference on Machine Learning, pp. 4413-4423 2019.
Alfredo Nazábal, Pablo M. Olmos, Zoubin Ghahramani, and Isabel Valera. Handling incomplete heterogencous data using vaes. Pattern Recognition, 107:107501, 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011. 1.
10

---

Published as a conference paper at ICLR 2022
Yuge Shi, N. Siddharth, Brooks Paige, and Philip H.S. Torr. Variational miixture-of-expee autoen- coders for multi modal deep generative models. arXiv, (NeurIPS), 2019. ISSN 23318422. URL
https://aaxiv: wwwppdt/////ooooo.... http:/pp//9911....11 03393. pd
Yuge Shi, Brooks Paige, Philip Torr, and Siddharth N. Relating by contrasting: A data-efficient framework for multimodal generative models. In ICLR 2021: The Ninth International Conference on Learning Representations. 2021.
N. Siddharth, T Brooks Paige, Jan- Willem Van de Meent, Alban Desmaison, Noah Goodman, Push- meet Kohli, Frank Wood, and Philip Torr. Learning disentangled representations with semi- supervised deep generative models. Advances in Neural Information Processing Systems, pp- 5925-5935, 2017.
Lewis Smith and Yarin Gal. Understanding measures of uncertainty for adversarial example detec- tion. arXiv preprint arXiv:/803. 08533,, 2018.
Thomas M. Sutter, Imant Daunhawer, and Julia E. Vogt. Multimodal generative learning utilizing jensen-shannon divergence. Workshop on Visually Grounded Interaction and Language at the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), volume 33, pp- 6100-6110 10, 2020.
Thomas Marco Sutter, Imant Daunhawer, and Julia Vogt. Generalized multimodal elbo. In ICLR 2021: The Ninth International Conference Learning Representations, 2021.
Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Joint multimodal learning with deep generative models. In International Conference on Learning Representations Workshop, 2016.
Jakub M. Tomezak and Max Welling. VAE with vampprior. Proceedings of Machine Learning Research, 2018.
Yao Hung Tsai, Paul Pu Liang Amir Ali Bagherzade, Louis-Philippe Morency, and Ruslan Salakhutdinov. Learning factorized multimodal representations. In International Conference on Learning Representations, 2019.
Ramakrishna Vedantam, Ian Fischer, Jonathan Huang, and Kevin Murphy, Generative models of visually grounded imagination. In International Conference on Learning Representations, 2018.
Welinder, Branson, Mita, C. Wah, Schroff, Belongie, and P. Perona. Caltech-UCSD California Institute of Technology, 2010.
Birds 200. Technical Report
Mike Wu and Noah Goodman. Multimodal generative models for scalable weakly-supervised learn- ing. Adv. Neural Inf. Process. Syst. DeNiseeeeeNNNiss55555. 2018. ISSN 10495258. URL https: /ops://asse //arxivavvvv... wttgp/d/ 05335. pddf

---

Published as a conference paper at ICLR 2022
A DERIVATION OF THE OBJECTIVE
The variational lower bound for the case when and t are both observed, following the notation in Figure 2. derives as:
/ po, s, z)dz
log po, p (t. log
/ log Pa.v (s, t,
1 (2 (2)t,s) uou y0 (z)t. 8)dz
Following Joy et al. (2021). assuming s LLtt and applying Bayes rule we have
go (t(()))),
l0.o (z|t, s)
where go (t|s) = (2)
gø.p (t|s)
which can be substituted into the lower bound to obtain
/ log
(s, 1,s,tt,)4u do (t/s) qu (z(s))a,, dz
log (t.s)
(o(z)s) (t/z)
4d. (t/s)
(t/z)
log (z(s 4o (z)s)
(2/s)
log lo.p (t|s) + ptg.(t).
(6)
Io. (t|s)
B EFFICIENT GRADIENT ESTIMATION
Given the objective in (6). note that the first term is quite complex, and requires estimating a weight ratio that involves additional integral for qo.o s). This has significant effect, the naive Monte-Carlo estimator of
(t z))
Po(s z)pp (z
log
(t [s) (t
(z s) 2))
-Epte) (
2)
log (s P)) t). (z 2)) s) (t z) lg.0 (t [s] Vo.p log
po(s 2 p (z t) (z 8) (t. x)
Vone
20
Lo
can be very noisy, and prohibit learning effectively. To mitigate this, we note that the first term computes gradients for the encoder parameters (Ø, (p) through a ratio of probabilities, whereas the second term does through log probabilities. Numerically, the latter
lot more stable learn simply drop the first term in ( by employing stop-gradien on the ratio (t/x)) We further support this change with empirical results (cf. Figure 10) that show how badly the signal-to-noise ratio (SNR) is affected for the gradients with respect to the encoder parameters. We further note that Joy by an empirical study, but where they detach the sampled z-we find that our simplification that detaches the weight itself works more stably and effectively.
from than the former, and
(t)s)
(202T) perform similar modification also motivated
Nose encoder patameters
Signil ration classifter parameters
Stap Gradkent Vanill
Stop Gradient
vanilla
5
300
200
300
200
Iterttonn
Figure 10: SNR for parameters (6. left) and (p. right). The blue curve denotes the simplified estimator using stop gradient, and the orange curve indicates the full estimator in (7). Higher values leads to improved learning.
12

---

Published as a conference paper at ICLR 2022
C WEIGHT SHARING
Another critical issue with naively training using (6), is that certain situations 9u (t z) struggles to learn features (typically style) for t, consequently making it difficult to generate realistic samples. This is due to the information entering the latent space only coming from s, which contains all of the information needed reconstruct s, but does not necessarily contain the information needed to reconstruct a corresponding t. Consequently, the term pa (s z) will learn appropriate features (like standard VAE decoder), but the term (t z) will fail to do so. In situations like this, where the information in t is nor subsumed by the information in there is no way for the model to know how to reconstruct a t. Introducing weight sharing into the bidirectional objective (2) removes this issue, as there is equal opportunity for information from both modalities to enter the latent space, consequently enabling appropriate features
learned the decoders po (s z) and pt z), which subsequently allow cross generations to be performed.
Furthermore, we also observe that when training with (2) we are able to obtain much more balanced likelihoods Table 2 In this setting we train two models separately using (6) with s = MNIST SVHN and s = MNIST respectively. At test time, we then 'flip'
and SVHN and then with the modalities and the corresponding networks, allowing direction. Clearly we see that we only obtain reasonable marginal likelihoods in the direction for which we train. Training with the bidirectional objective completely removes this deficiency, now introduce a balance between the modalities.
obtain marginal likelihoods in each
Table 2: Marginal likelihoods.
Train Direction
Test Direction
M, S=
S= =
Bi
s M, =
-14733.6
-00249.gtip
14761.3
s= ==
-428728.7
- -11668.1
-11355.4
D REUSING APPROXIMATE POSTERIOR MC SAMPLE
When approximating (t s) through MC sampling, we find that it is essential for numerical
stability to include the sample from the approximate posterior. Before considering why, we must first outline the numerical implementation of q s), which for samples 21:K ~ go(z s) is computed using the LogSumExp trick as:
K
log (t (t| S) 2 log
exp log qo (t/zs),
(8)
where the ratio (t)t)s) is computed as expflog z) lo s)}. Given that the (t(s)
LogSumExp trick is defined as:
log xn x² + log Σ exp(xn x²);
where x= = max(f1,.............. J. The ratio will be computed as
(t z))
(t (t = exp(log (t z) - log gr (t/z²) log
expllog a (t/za)
log d (t|z")]
(10)
where z² arg max log qu (t)zz), [t) For numerical stability, we require that log (t z) log (t|z"), otherwise the computation may blow up when taking the exponent. To enforce this, need to include the sample into the LogSumExp function, doing so will cause the first two terms to either cancel z = x" or yield a negative value, consequently leading to stable computation when taking the exponent.
13

---

Published as a conference paper at ICLR 2022
E EXTENSION BEYOND THE BI-MODAL CASE
Here offer further detail how MEME can be extended heyond the bi-modal case, i.e. when the number of modalities M > 2. Note that the central thesis in MEME is that the evidence lower bound (ELBO) offers an implicit way regularise different representations if viewed from the posterior- prior perspective, which can be used to build effective multimodal DGMs that are additionally ap- plicable to aorseell-observed data. In MEME, we explore the utility of this implicit regularisation in the simplest possible manner to show that direct application of this to the multi-modal setting would involve the case where M = 2.
The way extend, say for M = 3, involves additionally employing explicit combination for two modalities in the prior (instead of just 1). This additional combination could be something like mixture product, following from previous approaches. More formally, if were to denote the implicit regularisation between posterior and prior as R,(...). and an explicit regularisation function Re(-,-), and the three modalities m,, m2, and my, this would mean would compute
RR Rm,,, ma)) + Rg (m2, R.(m1 ma)) + Ra Rg mm)))
(11)
assuming that R. was commutative, as is the case for products and mixtures. There are indeed more terms to compute now compared to M = 2, which only needs R2 [,(m,, m2), but note that R2 is still crucial-it does not diminish because we are additionally employing R.-
As stated in prior work et al.) 2016 Wu & Goodman 2018 Shi et al. 2019), we follow the reasoning that the actual number modalities, at least when considering embodied perception, is not likely to get much larger, the increase in number of terms, while requiring more computa- tion, is unlikely to become intractable. Note that prior work on multimodal VAEs also suffer when extending the number of modalities in terms of the number of paths information flows through.
We do not explore this setting empirically as our priary goal is to highlight the utility of this implicit regularisation for multi-modal DGMs, and its effectiveness at handling oobseally-observed data.
F CLOSED FORM EXPRESSION FOR WASSERTEIN DISTANCE BETWEEN TWO GAUSSIANS
The Wassertein-2 22 distance between two probability measures μ and v on IR is defined as W2(u, v) = inf B[] x with ~ a and Y ~ v. Given μ = N(11, En) and v N (mm2, M2), the 2- 2-Wassertein is then given
d = =1mm + m2 Tr(E; + Σn Σ
For a detailed proof please see (Givens Shorti 2002).
G CANONICAL CORRELATION ANALYSIS
Following Shi et al. (2019): Massiceti et al. 2018), we report cross-coherence scores for CUB using Canonical Correlation Analysis (CCA). Given paired observations x1 E RT and x₂ E R2 CCA learns projection weights WT E R xk and W E R which minimise the correlation between the projections W x1 and WT X2. The correlations between data pair (x1, x2) can thus be calculated as
<(x)) (X2)
corr( (x1, corr(x1,
X2)
(12)
| [>>))
where ((x,) WXX,, (. x,).
Following Shi et al. 2019), we use feature extractors to pre-process the data. Specifically, features for image data generated from off- the-hhe-selff ResNet- 101 network. For text data, first fit a FastText model on all sentences, resulting in 300-d projection for each word Bojanowski et al. 2017), the representation then computed the average over the words in the sentence.
14

---

Published as a conference paper at ICLR 2022
9
3
3
6
8
1
/
/
8
&
2
6
6
6
7
7
7
7
2
2
2
6
7
C
6
9
4
8
7
&
3
8
7
4
9
O
7
E
5
5
8
I
3
9
4
7
7
5
2
9
7
O
4
V
2
3
6
4
6
6
6
S
5
A
O
5
9
9
3
O
3
6
3
6
7
7
6
6
G
9
S
7
4
2
10
5
3
7
3
7
7
9
9
7
7
5
9
5
9
8
&
3
8
6
7
5
/
y
8
7
Lo
7
9
4
3
2
3
4
8
7
9
5
Figure 11: MNIST + SVHN (Left) and SVHN
MNIST (Right). for the fully observed case.
9
2
3
5
8
1
/
9
6
8
2
6
6
6
7
7
7
7
5
7
7
2
1
2
2
6
/
7
7
T
F
&
9
5
6
6
O
0
6
6
/
9
O
6
C
3
15
/
I
Y
2
8
7
3
3
3
4
2
Q
Y
&
2
7
4
9
4
6
Y
3
3
5
2
3
9
Y
5
9
7
7
7
9
7
7
O
4
6
7
5
5
L
9
5
S
O
5
6
/
9
3
4
7
T
7
6
6
6
E
9
5
6
2
/
4
2
6
6
9
10
4
3
7
9
7
&
C
7
8
4
2
7
2
6
9
6
3
2
6
5
4
2
7
5
2
5
2
3
8
8
6
7
6
3
Figure 12: MNIST + SVHN (Left) and SVHN + MNIST (Right), when SVHN is observed 50% of the time.
9
2
8
9
2
6
7
7
7
7
5
7
7
2
6
7
37
3
9
O
0
9
6
3
/
/
Y
O
2
7
3
3
3
8
4
3
7
O
5
?
9
5
S
&
2
7
4
9
4
O
3
O
6
O
/
7
Z
3
3
5
D
3
2
2
2
7
O
H
5
8
9
3
9
4
#
7
7
5
5
2
9
7
O
O
O
+
V
2
6
7
&
6
S
5
3
S
9
7
5
7
6
G
9
S
7
4
Z
6
9
10C
O
5
5
3
7
3
7
3
9
2
6
9
V
/
7
3
2
6
3
7
5
7
/
5
/
F
9
5
7
2
6
2
9
0
&
R
6
9
6
9
8
C
5
6
y
8
7
6
0
6
3
2
3
4
9
Figure 13: MNIST SVHI (Left) and SVHN
MNIST (Right), when MNIST'S observed 50% of the time.
15

---

Published as a conference paper at ICLR 2022
9
3
3
8
8
&
2
7
7
7
7
7
2
2
7
9
O
6
/
9
O
2
8
9
3
3
8
4
S
%
2
7
4
7
4
O
9
O
2
3
3
S
2
5
6
S
O
5
9
7
&
60
L
Z
7
5
2
9
7
O
O
3
7
G
5
S
O
5
7
5
7
G
9
O
5
0
7
4
2
10
5
3
7
3
7
6
3
1
2
6
9
/
7
6
3
9
P
6
7
9
2
7
7
5
/
F
9
5
2
9
3
7
5
L
9
&
6
2
4
4
4
S
2
6
7
6
9
6
3
2
9
8
la
4
Figure 14: MNIST
SVHN (Left) and SVHN
MNIST (Right), when SVHN observed 25% the time.
9
3
#
3
5
8
/
/
31
4
5
3
9
1
3
&
2
6
4
6
7
7
7
7
7
8
5
8
3
5
3
7
7
2
2
6
M
7
6
C
6
O
2
8
3
3
8
4
5
9
S
4
7
4
O
6
6
O
Y
T
Z
3
3
&
5
2
3 4
Y
5
8
9
3
9
6
7
5
2
9
7
O
O
4
V
2
3
5
6
4
7
6
6
5
5
A
O
9
3
S
O
3
6
9
9
3
4
7
7
6
ON
OF
G
9
O
7
4
2
100
3
7
D
7
6
7
3
7
7
5
9
S
7
5
7
9
&
5
5
10
2
1
y
7
7
6
9
6
3
2
3
9
7
Figure 15: MNIST
SVHN (Leff) and SVHN + MNIST (Right), when MNIST is observed 25% of the time.
9
2
4
3
5
8
1
/
9
S
G
9
8
2
6
6
7
7
7
7
6
9
0
7
7
2
1
2
2
6
/
7
7
O
7
7
&
9
S
6
6
6
O
6
6
9
5
6
G
G
O
3
15
/
9
(
+
0
2
R
7
3
8
4
O
6
&
3
Y/
3
6
&
2
7
4
4
O
6
6
CO
4
/
S
V
7
3
3
4
5
2
3
2
SOG
O
Y
5
9
3
9
7
7
2
6
7
O
O
Y
3
G
7
5
5
/
9
3
S
O
3
9
5
4
7
5
7
8
5
9
5
0
7
/
4
6
3
10
O
5
S
7
4
3
7
3
A
A
4
5
7
7
9
3
9
2
6
9
/
7
6
3
7
6
3
7
5
/
f
9
5
2
2
3
6
2
3
9
C
0
8
5
9
5
9
>
7
2
6
Figure 16: MNIST
SVHN (Left) and SVHN - MNIST (Right), when SVHN is observed 12.5% of
time.
16

---

Published as a conference paper at ICLR 2022
9
2
3
5
8
1
8
7
5
&
2
6
6
6
7
7
7
7 19
6
2
7
2
2
/
7
9
5
6
6
O
6
/
9
9
4
8
7
&
3
8
4
3
3
S
Y
4
O
2
3
F
&
5
3
&
7
7
O
3
7
5
9
5
5
5
O
9
5
7
5
7
&'
7
7
6
9
O
5
0
2
/
4
2
6
7
10
C
5
4
3
7
9
7
A
2
4
4
9
2
6
9
/
7
6
16
3
4
2
6
4
7
5
5
2
5
C
O
8
22
5
9
6
9
6
3
2
7
Figure 17: MNIST -> SVHN (Left) and SVHN -> MNIST (Right), when MNIST is observed 12.5% of the
time.
Table 3: Coherence Scores for MNIST + SVHIN (Top) and for SVHIN + MNIST (Bottom). Subscript indicates which modality is always present during training, f indicates the percentage of matched samples. Higher is better.
MNIST SVHN
Model
f= 1.0
f f= 0.5
f 0.25
f 0.125
f f= 0.0625
MEMESvHN
0,625 1 0.007
0.551 1 0.008
0.323 1 0.025
0.172 1 0.016
0.143 1 0.009
MMVAESVER
0.581 = 0.008
MVAESVHN
0.123 0.003
0.110 0.014
0.112 + 0.005
0.105
0.005
0.105 + 0.006
MEMEXNIST
0.625 + 0.007
0.572 + 0.003
0.485 + 0.013
0.470
0.009
0.451 + 0.011
MMVAEMNIST
0.581 = 0.008
MVAFMNIST
0.123 + 0.003
0.111 + 0.007
0.112 + 0.013
0.116
0.012
0.116 + 0.005
MEMESPLI
0.625 + 0.007
0.625 + 0.008
0.503 + 0.008
0.467
0.013
0.387 + 0.010
MVAEsPLIT
0.123 + 0.003
0.108 + 0.005
0.101 + 0.005
0.101 0.001
0.101 + 0.002
SVHN + MNIST
Model
==0
f 0.5
f 0.25
f= 0.125
f 0.0625
MEMESVEN
0.752 = 0.004
0.726 + 0.006
0.652 + 0.008
0.557
0.018
0.477 + 0.012
MMVAE ESEAEN
0.735 0.010
MVAESVIIN
0.498 = 0.100
0.305 + 0.011
0.268 + 0.010
0.220 = 0.020
0.188 + 0.012
MEMEMNIST
0.752 0.004
0.715 + 0.003
0.603 : 0.018
0.546
0.012
0.446 + 0.008
MMVAFMNIST
0.735 + 0.010
MVAFMNNIT
0.498 + 0.100
0.365 + 0.014
0.350 + 0.008
0.302 + 0.0155
0.249 + 0.014
MEMEsELIT
0.752 + 0.004
0.718 + 0.002
0.621 + 0.007
0.568
0.014
0.503 + 0.001
MVAEsPLIT
0.498 + 0.100
0.338 + 0.013
0.273 + 0.003
0.249 + 0.019
0.169 + 0.001
H ADDITIONAL RESULTS
H.1 MVAE LATENT ACCURACIES
The superior accuracy in latent accuracy when classifying MNIST from MVAE is due to a com- plete failure to construct a joint representation, which is evidenced in its failure to perform cross- generation. Failure to construct joint representations aids latent classification, as the encoders just learn to construct representations for single modalities, this then provides more flexibility and hence better classification. In/Figure 19 we further provide a t-SNE plot to demonstrate that MVAE places representations for MNIST modality in completely different parts of the latent space to SVHN. Here
17

---

Published as a conference paper at ICLR 2022
Fully observed. a grey bird with darker
this is white bird with wings and and black beak.
brown mixed in and a short brown beak.
yellow bird with black and white wings with a black beak.
small brown bird with white belly.
Captions observed 50% of the time.
than a particular winge a with and wings has on yellow.
the bird has two large grey kingbars and orange feet.
yellow bird with black and white wings with a black beak.
below amall has has that bill yel low and.
Images observed 50% of the time.
blacks this bird has has that are that and has yellow belly.
tiny brown bird with white breast and short stubby bill.
bird this bird has red with bird with and pointy.
this is yellow bird with black
its head.
Captions observed
25% of the time.
this is a puffy bird with a bright yellow chest with white streaks along the feathers.
throat is is yellow with yellow gray chest has medium belly belly throat.
bird green small has crown as wing crown white white abdonen crown and.
this bird has a small bill with a black head and wings white body.
crest this Images observed 25% of the time.
looking with with with
the bird had large white breast in <exc> to head size.
and face body.
this bird has a black belly breast and head gray and white wings and red and feez.
rotund white bi11 large bird brown black back mostly with with breast flying.
Captions observed 12.5% of the time.
the bird wings a is and nape the and crown rectrices grey beak its and tail.
white bel y with brown body and very short small brown beak.
a bird with a shart rounded beak which ends in point stark white eyes and white threat.
red this bird skinny black black red feet beak downwards short and a.
Images observed 12.5% of the time.
an a is colerful bird short with and black and small short orange light small light over. this is white all with flat beak and black 's curved for light has body is.
small bird with a long beak and blue wing feathers with brown body.
this is a large black bird with a long neck and bright orange cheek patches.
Figure 18: MEME eross-modal generations for CUB.
we can see that representations for each modality are completely separated, meaning that there is no shared representation. Furthermore, MNIST is well clustered, unlike SVHN. Consequently it is far casier for the classifier to predict the MNIST digit as the representations do not contain any information associated with SVHN.
18

---

Published as a conference paper at ICLR 2022
Table 4: Latent Space Linear Digit Classification.
MNIST
Model
1.0
0.5
0.25
0.125
0.0625
MEMEsvIN
0.908 + 0.007
0.881 + 0.006
0.870 + 0.007
0.815 + 0.005
0.795 + 0.010
AMVAEsVHN
0.886 + 0.003
MVAEşvHN
0.892 1 0.005
0.895 + 0.003
0.890 + 0.003
0.887
0.004
0.880
0.003
OUrSMNIST
0.908 + 0.007
0.882 1 0.003
0.844 1: 0.003
0.824 + 0.006
0.807 1 0.005
MMVAEXNIST
0.886 + 0.003
MVAExNIST
0.892 + 0.005
0.895
0.002
0.898
0.004
0.896 + 0.002
0.895
0.002
MEMESPUI
0.908 + 0.007
0,914
0.003
0.893 + 0.005
0.883 + 0.006
0.856
0.003
MVAEsPLIT
0.892 + 0.005
0.898
0.005
0.895 + 0.001
0.894 + 0.001
0.898 0.001
SVHN
Model
1.0
0.5
0.25
0.125
0.0625
MEMESvHN
0.648 + 0.012
0.549 + 0.008
0.295 + 0.025
0.149 + 0.006
0.113 + 0.003
MMVAEsvHN
0.499 1 0.045
MVAEsvHI
0.131 + 0.010
0.106
0.008
0.107 1 0.003
0.105 + 0.005
0.102
0.001
OUESMNIST
0.648 + 0.012
0.581
0.008
0.398 + 0.029
0.384 + 0.017
0.362 0.018
IMMAEASIIST
0.499 + 0.045
MVAEMNIST
0.131 + 0.010
0.106 + 0.005
0.106 + 0.003
0.107 + 0.005
0.101 0.005
MEMEsPLIT
0.648 + 0.012
0.675
0.004
0.507 + 0.003
0.432
0.011
0.316
0.020
MVAEsPLIT
0.131 + 0.010
0.107 + 0.003
0.109 + 0.003
0.104 + 0.007
0.100 + 0.008
Table 5: Correlation Values for CUB cross generations. Higher better.
Image Captions
Model
GT
f f= 1.0
f = 0.5
f == 0.25
= 0.125
MEM Eamage
0.106 1 0.000
0.064 1: 0.011
0.042 +: 0.005
0.026
0.002
0.029 +: 0.003
MM VAV Emmage
0.106 + 0.000
0.060 + 0.010
M AVAEmaage
0.106 = 0.000
00.02 + 0.001
-0.000 0.004
0.001 + 0.004
-0.001 = 0.005
MEME Caytions
0.106 + 0.000
0.064 + 0.011
0.062 + 0.006
0.048 + 0.004
0.052 + 0.002
MMVAE.C Captioon
0.106 = 0.000
0.060 = 0.010
M Capptions
0.106 + 0.000
-0.002 + 0.001
-0.000 0.004
0.000
0.003
0.001 + 0.002
MEM
0.106 + 0.000
0.064 + 0.011
0.046 + 0.005
0.031 + 0.006
0.027 - 0.005
EVAESPPIT
0.106 0.000
-0.002 + 0.001
0.000 + 0.003
0.000 0.005
-0.001 0.003
Caption + Image
Model
GT
f f= 1.0
f= 0.5
f 0.25
f 0.125
MEM Eimage
0.106 + 0.000
0.074 + 0.001
0.058 + 0.002
0.051 + 0.001
0.046 = 0.004
MMVAE Exmage
0.106 = 0.000
0.058 = 0.001
M AVAEEmage
0.106 = 0.000
-0.002 0.001
-0.002
0.000
-0.002 + 0.001
-0.001 = 0.001
CurSCuptions
0.106 = 0.000
0.074 + 0.001
0.059 + 0.003
0.050 0.001
0.053 = 0.001
MMVAF Captions
0.106 + 0.000
0.058 0.001
MVA ECaptions
0.106 + 0.000
0.002 + 0.001
-0.001
0.002
-0.003 + 0.002
-0.002 + 0.001
MEMESPLI
0.106 0.000
0.074 0.001
0.061 + 0.002
0.047
0.003
0.049
0.003
MVAESPLIT
0.106 + 0.000
-0.002 + 0.001
-0.002
0.002
-0.002 + 0.001
-0.002 = 0.001
H.2 GENERATIVE CAPABILITY
We report the mutual information between the parameters w a pre-trained classifier and the labels y for a corresponding reconstruction x. The mutual information gives an indication of the amount of information we would gain about w for a label y given x, this provides an indicator to how our- of-distribution x is. If x is realistic reconstruction, then there will be low MI, conversely,
19

---

Published as a conference paper at ICLR 2022
Latent Veriable T.SNE
40
54
20-
57
m4
Figure 19; T-SNE plot indicating the complete failure of MVAE construct joint representations. indicates SVHN (low transparency), m indicates MNIST (high transparency).
un-realistic x will manifest as a high MI as there is a large amount of information to be gained about w. The MI for this setting is given
Ty, w x, D) = Hlp(y x, D)) PP((DD)
[Hp(y) x, xxw)]]-
Rather than using dropout Gal (2016); Smith Gal (2018) which requires an ensemble of multiple classifiers, we instead replace the last layer with a sparse variational GP. This allows us to esti- mate pyy x, D) I ply xr w p(w D)dw using Monte Carlo samples and similarly estimate Ep(w(D) [ply []]]]. We display the MI scores in Table 6) where we see that our model is able
to obtain superior results.
Table 6: Mutual Information Scores. Lower is better.
MNIST
Model
1.0
0.5
0.25
0.125
0.0625
OurSsvin
0.075 + 0.002
0.086 + 0.003
0.101 + 0.002
0.102 + 0.004
0.103 + 0.001
MMVAEsvIN
0.105 + 0.004
MVAEsvHI
0.11 0.00551
0.107 1: 0.007
0.106 +: 0.004
0.106
0.012
0.142 1: 0.007
OursMNIST
0.073 + 0.002
0,087 +: 0.001
0.101 +: 0.001
0.099
0.001
0.104 + 0.002
MMVAEMNIST
0.105 0.004
MVAEMNIST
0.11 0.00551
0.102 + 0.00529
0.1 + 0.00321
0.0117
0.0927
0.00709
MEMFsELIT
0.908 0.007
0.914
0.003
0.893 + 0.005
0.883 0.006
0.856 + 0.003
MVAESELIT
0.11 0.00551
0.104 = 0.006
0.099 + 0.003
0.0117
0.098 + 0.005
SVHN
Model
1.0
0.5
0.25
0.125
0.0625
OurssvIN
0.036 + 0.001
0.047 = 0.002
0.071 + 0.003
0.107
0.007
0.134 + 0.003
MMVAESviN
0.042 0.001
MVAEsvIN
0.163 0.003
0.166 0.010
0.165 1: 0.003
0.164 0.004
0.176 +: 0.004
OurswNIsT
0.036 0.001
0.048 0.001
0.085 + 0.006
0.111 0.004
0.142 + 0.005
MMVAFMNIIT
0.042 + 0.001
MVAEMNIST
0.163 + 0.003
0.175 + 0.00551
0.17 + 0.0102
0.174 0.012
0.182 + 0.00404
MEMESPLIT
0,648 0.012
0.675
0.004
0.507 1 0.003
0.432
0.011
0.316 1 0.020
MVAESFLIT
0.163 + 0.003
0.165 + 0.01
0.172 1 0.015
0.173 0.013
0.179 1 0.005
H.3 T-SNE PLOTS WHEN PARTIALLY OBSERVING BOTH MODALITIES
In Figure 20 we see that partially observing MNIST leads to less structure in the latent space.
20

---

Published as a conference paper at ICLR 2022
T-QNE
Veriable
PG
40-
-30
20
40
-46
Figure 20: f= 0.25, Left) t-SNE when partially observing MNIST. Right) t-SNE when partially observing SVHN.
Table Coherence Scores MMVAE using Laplace posterior prior.
MNIST
SVHN
91.8%
65.2%
I MMVAE BASELINE WITH LAPLACE POSTERIOR AND PRIOR
The difference in results between our implementation of MVAE and the ones in the paper (Shi al. (2019), becuase we restrict MEME to use Gaussian distributions for the posterior and prior, and therefore adopt Gaussian posteriors and priors for all three models to ensure like-for-like comparison. Better results for MMVAE can be obtained by using Laplace posteriors and priors, and In Table 7 we display coherence scores using our implementation of MMVAE using Laplace posterior and prior. Our implementation inline with the results reported in Shi et al. (2019) indicating that the baseline for MMVAE is accurate.
J ABLATION STUDIES
Here carry out two ablation studies to test the hypotheses: 1) How sensitive is the model to the number of pseudo samples in À and 2) What the effect of training the model using only paired data for given fraction of the dataset.
J.1 SENSITIVITY NUMBER PSEUDO-SAMPLES
In Figure 21 we plot results where the number of pseudo samples is varied for different obser- vation rates. Ideally we expect to see the results decrease in their performance as the number of pseudo-samples is minimised. This is due to the number of components being present in the mixture - (2) Σ (o(z) uf)) also being decreased, thus reducing the its ability approximate the sensitive, due a higher dependence on the prior approximation, and a higher number of pseudo samples typically
true prior p(z) (0 (z tt)((t)dt, dtt. As expected lower observation rates
leads to better results.
J.2 TRAINING USING ONLY PAIRED DATA
Here we test the models ability to leverage partially observed data to improve the results. If the model is successfully able to leverage the partially observed samples, then we should see a decrease in the efficacy if we train the model using only paired samples, i.e. model trained with 25% paired and 75% partially observed should perform improve the results over a model trained with only the 25% paired data. In other words we omit, the first two partially observed terms in (5), discarding D, and D.- In Figure 22 we can see that the model is able to use the partially observed modalities to improve its results.
21

---

Published as a conference paper at ICLR 2022
Latent HNST
Latent SUHN
1.0
NAETT two
SVML 1-20
0.7
Quin 1-3.:35
SPUJJ-0.123
0/5-
0.4
7VNN N-0.220
0.3
FRUT 2r0.125
07-
Phoudo Samples
Mum Fhoaao Samples
Conerence Batween ST and SV
Cchorance Betwean SVHH and MIUST
0.00
0.75
HH IISL
0.3
RMST 5m
0.50
SPLIT 6.122
mast 125
0.50
SAN 1-3.3
0.2
5ANV 1-2::25
0.15
Figure 21: How performance varies for different numbers of psuedo samples. Number pscudo samples ranges from 1 to 100 on the x axis.
Latend Acturacy MNST
Latient Accuracy SVMIN
MEML_ SN 51
MEHE MM12
Hotte 3VMN
PEFRL 3/HE
NENE_SPUT
MENE
0.8
0.5
0.4
0.2625
0.25
0/5525
912: Ctreaation footion
(bsenotinn
Coherence Between Mb: ST and 5V
Cehecaece Betowen SVL and WUJCT
HENE REME_YVHN
HEHE HUST
HENE_5P SPUT
HEHE SP.T"
03
0.4
00075
desrvationntaatte
Cessrvation frction
Figure 22: How performance varies when training using only fraction of the partially observed data.
22

---

Published as a conference paper at ICLR 2022
K TRAINING DETAILS
The architechtures are very simple and cean easily be implemented in popular deep learning frame- works such as Pytorch and Tensorflow. However, we do provide a release of the codebase at the following location: https:/giihub..om///ivvyy/neee
MNIST-SVHN We provide the architectures used in Table 8b and Table 8a We used the Adam optimizer with learning rate of 0.0005 and beta values of (0.9, 0.999) for 100 epochs, training consumed around 2Gb of memory.
CUB We provide the architectures used in Table 8c and Table 8d We used the Adam optimizer with a learning rate of 0.0001 and beta values of (0.9, 0.999) for 300 epochs, training consumed around 3Gb of memory.
Encoder
Decoder
Input e R1r28125
Input RR4
FC. 400 ReLU
FC. 400 ReLU
FC. I. FC. L
FC. x 28 28 Sigmoid
(a) MNIST dataset.
Encoder
Input 1r28r28
4x4 cony. 32 stride 2 pad & ReI.U
4x4 conv. 64 stride 2 pad I & ReLU
4x4 conv. 128 stride 2 pad & ReLU
4x4 cony. stride pad 0, 4x4 conv. stride pad
Decoder
Inpul & R²
4x4 upconv. 128 stride I pad 0 & ReLU
4x4 upconv, stride pad & ReLU
4x4 upconv. stride pad & ReLU
4x4 upconv. 3 stride pad & Sigmoid
(b) SVHN dataset.
Encoder
Decoder
Input c: 2020
Input E: R+
FC. 1024 ELU
FC, ELU
FC. 512 ELU
FC. ELU
FC. 256 ELU
FC. 1024 ELU
FC. FC. L
FC. 2048
(c) CUB image dataset.
Encoder
Input E 11500
Word Emhh 256
4x4 conv. 32 stride pad & BatchNorm2d & ReLU
4x4 conv. 64 stride 2 pad & BatchNorm2d & ReLU
4x4 conv. 128 stride pad & BatchNorm2d & ReLU
1x4 conv. 256 stride 1x2 pad 0x1 & BatchNorm22 & ReLU
1x4 conv. 512 stride 1x2 pad & BatchNorm2d & ReLU
4x4 stride pad 0, 4x4 I. stride pad
Decoder
Input E: R²
4x4 upconv. stride 1 pad ReLU
1x4 upconv. 256 stride 1x2 pad 0x1 & BatchNorm2d ReLU
1x4 upconv. 128 stride 1x2 pad 0x1 & BatchNorm2d & ReL.
4x4 upconv. 64 stride 2 pad 1 & BatchNorm22 & ReLU
4x4 upconv. 32 stride 2 pad 1 & BatchNorm2d & ReLU
4x4 upconw. stride pad & ReL.U
Word Emb."" T 1590
(d) CUB Langu-Lee dataset.
Table 8: Encoder and decoder architectures.
23


Reviews:
[{"review_id": "vQWfd724zf", "reviewer": "Reviewer_ag7M", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is very well organized and easy to read.\n- The proposed MEME is a straightforward extension of an existing semi-supervised VAE to a multimodal setting, which is easy to understand and can be easily adapted to partial train settings.\n- Experiments confirm the effectiveness of MEME compared to MVAE and MMVAE in both partial and complete data settings.\n\nWeakness:\n- Since MEME is an extension of existing semi-supervised models, it is understandable that MEME performs better than MVAE in partial settings. However, why does MEME, which is just an extension of semi-supervised VAEs, perform better than MVAE and MMVAE even in the complete setting? What parts of MEME are important to achieve better results than them? In other words, the authors discuss the merits of MEME only from a semi-supervised perspective, and therefore should explicitly discuss why it is good in the complete setting. Also, the authors do not explain why MEME shows a better trend than MVAE or MMVAE in the relatedness experiments.\n- I understand that the authors are specifically focusing on the bimodal case in this paper. However, traditional explicit combinations, such as mixture and products, are intended to combine more than two modalities. In the last part of section 3.3, the authors state that an explicit combination can be added on top of the implicit combination to handle cases beyond two modalities (I don't understand the details of this method. Does this mean that the bimodal representations by the implicit combination are combined with the explicit one?).  However, this approach is a combination of implicit and explicit methods, so it does not mean that implicit itself can be extended to more than two modalities. Therefore, this implies that implicit combinations are essentially only applicable to bimodal. It is true that early multimodal VAEs such as JMVAE only considered bimodal, but unlike MEME, it does not mean that it cannot be extended to more than two modalities, although it requires exponentially increasing memory cost. Therefore, the fact that MEME is essentially not scalable to more than two is a fatal problem in the study of multimodal VAEs, which is nevertheless underestimated and hardly discussed in this paper.\n- Even though MoPoE is an extension of MVAE and MMVAE and has been shown to perform better than them, why hasn't it been compared in experiments?", "summary_of_the_paper": "This paper proposes MEME as a new method of multimodal VAE, which is an extension of semi-supervised VAEs and can handle partial train settings. Experimental results show that the proposed method outperforms the conventional methods, MVAE and MMVAE, in both partial and complete settings. Furthermore, the proposed method shows an interesting trend in its ability to capture the relatedness compared to these.", "summary_of_the_review": "The MEME proposed by the authors in this study achieves high performance in both partial and complete cases. It also shows interesting results in terms of relatedness. However, there is a lack of discussion on why MEME is better than existing methods and an explanation of its limitations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "te3rDj5raTv", "reviewer": "Reviewer_1avA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1) The main strength of this paper lies in its novel formulation of multimodal VAES and it avoids the need for explict specifications such as products and regularizers to combine information across modalities. The proposed approach also allows the joint distribution to be modeled under the partially-observed setting, where a percentage of observations have missing modalities.  The approach is also intuitive and well-detailed such that it is easily understandable.\n\n2) The equations are easy to understand and constructing a mirrored version of the VAE by swapping the modalities as well as their corresponding parameters is a clever idea that is premised on  the insight of finite conditional entropies. \n\n3) The authors empirically prove the effectiveness of their proposed approach through extensive experiments and ablations on the \nMNIST-SVHN and CUB datasets. \n\nWeaknesses: \n\n1) The reasons for confining the focus of this work to two modalities is completely understandable. However, given the fact that it works well under the partially observed setting, it will be insightful to expand the discussion on generalizing beyond two modalities. With the recent focus on representation learning from multiple (more than 2) modalities, it might be helpful to discuss more if it's possible to extend the implicit combination to multiple modalities instead of using explicit combinations.\n\n2) It may also be helpful to provide more explanation on the intuition underlying the use of learnable pseudo samples to estimate the prior for learning from partially observed data.\n\n3) Minor point: the current results in the main paper are reflected through figures. It would be nice to have absolute numbers to compare in the main paper. For example, table 3 could be included in the main paper to highlight the effectiveness of the approach, despite the space constraints.", "summary_of_the_paper": "This paper presents an approach to model the joint distribution over data from different modalities such as vision and language. It seeks to address a limitation that is common to prior work on multimodal VAEs, which have typically combined information across modalities by using explicit methods including products, mixtures and their combinations.  Specifically, this work proposes to exploit mutual supervision between different modalities to circumvent the need for the above-mentioned explicit combinations. Additionally, it provides a natural and intuitive extension to learning from partially observed data. The authors empirically prove the effectiveness of their proposed approach over prior work under both partial and complete data observation settings.", "summary_of_the_review": "In summary, this paper presents an interesting and intuitive approach for modeling the joint distributions across modalities. In particular, it appears to work well in the partially observed setting. Given the nature of real-world data which is usually very noisy, this is significant. Coupled with informative figures and extensive experiments (including those in the supplementary material), this paper could serve as a very useful reference for modeling the joint distribution of all modalities concurrently. Hence, I recommend acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "e-c_j8rVoB2", "reviewer": "Reviewer_ELno", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n1. The paper is well written and this reviewer enjoyed reading it. \n\n2. The proposed approach is solid and interesting\n2. The authors show rich experiments and analyses, including ablation studies and relatedness. Also, they ran the same models multiple times with different seeds and showed the deviation on the plots. \n\n\nWeaknesses\n\n1. In “Generalisation beyond two modalities” (on page 5), the authors say it is quite straightforward to extend MEME to be used for more than two modalities. Still, this reviewer can imagine it would be quite a complicated system with more than two modalities. Can you explain more details about the extension?\n\n2. Unclearness in Section 4. \n\n2-1. In Section 4, the notations with subscripts are confusing. Does the subscript SPLIT indicate when equal amounts of unpaired data from two modalities (e.g. MNIST and SVHN) are used?\n\n2-2. Although the authors said in Sec. 4.2 that the MEME contains rich class information from inputs, in Figure 7, the left plot does not support that the MEME models have a stronger representative power than the MVAE models, when f < 1, except the MEME_SPLIT model. This reviewer is unsure why the MEME models that show a high performance in the cross coherence metric shows lower performance in the classification task. \n\n2-3. In Figure 7 (left), MVAE_SPLIT performs better when less observation is available than seeing the entire data. This phenomenon also happens in MEME_SPLIT in the Latent Accuracy SVHN plot. Why?\n\n2-4. In Figure 8 (bottom), the orange portion seems larger than the blue portion in the MEME plot, while in the other two plots at the bottom, the numbers of unpaired and paired samples seem equal. This review is curious if the different settings were used for the MEME models. \n\n2-5. In Figure 9 (top, right), the MMVAE model completely fails to extract class information. However, this result is not coincident with the results from Figure 5, which shows a lot higher scores than MMVAEs’\n\n\n3. Typos. \n\n3-1. On page 3 (right before Sec. 3.2) Θ = {θ, θ} ← there are two thetas. \n\n3-2. In Figure 6, the subscripts are {Caption, Sentence, SPLIT}. Doesn't either Caption or Sentence have to be replaced with Image?\n\n\n4. The paper introduced a novel approach to the mirrored semi-supervised VAE by adding two additional regularizations to solve the missing unpaired data problem in the multimodal setting. However, the technical novelty is limited, except for extending the work of a semi-supervised VAE [Joy et al. (2021)].  Also, the problem itself is not new as the multimodal generative models were introduced in prior work [Wu et al. (2018)] and [Shi et al. (2019)]. ", "summary_of_the_paper": "The paper solves the multimodal problem in the partially-observed setting. With the observation that no one modality has all the information, the authors proposed a novel approach called MEME, which symmetrizes the semi-supervised VAE formulation by constructing a mirrored version. The proposed approach was demonstrated on standard metrics (cross coherence, latent accuracy) for multimodal VAEs across both partial and complete data settings (MNIST-SVHN, CUB) and shows that it outperforms prior work on both.", "summary_of_the_review": "The paper is about the new approach to solve the missing unpaired data problem in the multimodal setting. Although the approach is solid and interesting, it is incremental to the prior work. Some of the experiments demonstrating the superiority of the proposed approach are also unclear.   ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ma7SvCDBJuI", "reviewer": "Reviewer_WKus", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The presented framework (MEME) is an interesting and technically sound method for multi-modal data modeling. The paper is well-structured and easy to follow.\n2. The authors conduct extensive experiments and employ many different metrics to demonstrate the effectiveness of the methods. The experimental results are good enough to make a contribution to multi-modal VAEs. \n3. The method can handle partial data modalities. Even though the idea behind is not novel (like the partial label of semi-supervised VAEs), I still consider it as a contribution, since prior work usually handles the partially-observed modalities in a hard way (zero-padding or discarding). \n\nWeakness:\n1. The method extends the CCVAE which models the dependency between label and data. The information in labels is definitely not comparable to the information in a data modality. I agree that mutual supervision can mitigate this issue as it establishes a symmetric information flow. However, in qualitative results (Fig.12 - Fig.17) it is quite obvious that the generated samples of SVHN (left) are limited in certain colors (grayish) and shapes (thin and centered), while MNIST generations (right) contain much more variations. Can the authors provide more details on the architecture and parameters related to each modality? Can more powerful decoders for SVHN help generate more realistic SVHN samples? \n\n2. I'm a bit concerned about the scalability of the method. Even though the authors provide some potential ways to scale the current bi-modal setting to a multi-modal setting. Unlike other multi-modal VAEs that simply concatenates a new modality, MEME is based on a bi-modal setting. Thus three modalities will already need 6 information flows (e.g. for ($a,b,c$) modalities, $a \\rightarrow b, b \\rightarrow a, a \\rightarrow c, c \\rightarrow a, b \\rightarrow c, c\\rightarrow b$). Could the authors elaborate more on how to scale MEME to modalities > 2? The authors can also add some results on 3-modal datasets like MNIST-SVHN-Text used in [1] to compare the performance and training time of MEME and other baselines. \n\n\nMinor comments:\n1. The authors need to double-check the manuscripts --- some typos I can find may confuse the audience: 1) In  Fig.6 MEME_Caption should be MEME_Image? 2) in Fig.9 caption the middle is MMVAE, not MVAE? \n\n2. I'm curious about the comb-like distribution of Wassertein distance in Fig.8 SVHN-MNIST. Do authors have any intuition or comments on it? \n\n[1] Thomas M. Sutter, Imant Daunhawer, Julia E. Vogt, Multimodal Generative Learning Utilizing, Jensen-Shannon-Divergence, NeurIPS 2021", "summary_of_the_paper": "The authors propose a novel approach based on Variational Autoencoders to model the joint distribution of heterogeneous data (perceptually multi-modal, e.g. vision + language). Unlike prior work that typically handles idiosyncratic modalities with the explicit combination (concatenation or factorization), this proposed method implicitly introduces a dependency between two modalities via a prior regularization. The method extends CCVAE (Joy et al. 2021) by replacing the label as another modality. To compensate for the information disparity from the label to a data modality, the authors present a \"mutual supervision\" that uses a bi-directional information flow. The method can also train with partially-observed data where some modalities are missing. ", "summary_of_the_review": "This paper presents a method extending CCVAE from label and data to multiple modalities. The way of CCVAE handling semi-supervised can be used for training with partially-observed data. Even though the novelty is somewhat limited in VAE literature, it's still a contribution to multi-modal generative modeling. Thus I vote for a borderline acceptance at this point.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: PAPER: This paper introduces a new method to learn joint representations from multimodal data, with potentially missing data. The primary novelty builds from the idea of semi-supervised VAE, introducing the concept of bi-directional information flow, which is termed “mutual supervision”. This approach brings the same advantages of semi-supervised VAE to the multimodal setting, allowing the cross-modal interactions to be modeled in the latent space. 
DISCUSSION: The discussion brought many important issues, addressed by both reviewers and authors. In general, it seems that most reviewers appreciate the technical novelty of the paper, related to the mutual supervision. While some concerns were expressed about the similarity with semi-supervised VAE (Joy et al., 2021), I would agree with other reviewers and the authors that the extension is not straightforward. Bi-directional information flow is a worthwhile novelty in itself. One reviewer also mentioned a concern about previous work on multimodal generative models; previous work on the same topic should not preclude new papers, as long new technical ideas are proposed. The final observation is about modeling more than 3 modalities. This is effectively a challenge with the proposed idea and should be acknowledged in the paper, but it is also an issue for many other approaches. New research will be needed to study 3+ modalities, but it should be seen as a future work direction.
SUMMARY: Based on the reviews, discussion and personal reading of the paper, I lean towards acceptance of this paper. The paper introduces a new technical idea (bi-directional information flow, aka mutual supervision) which enables multimodal representation learning with missing data. The authors should revise their paper to acknowledge potential limitations of the approach (e.g., complexity challenges with 3+ modalities), but the idea is very interesting and worth publication.
Decision: Accept (Spotlight)