Multimodal Variational Autoencoders for Semi-Supervised Learning: In Defense of Product-of-Experts
Authors: Svetlana Kutuzova
Oswin Krause
Douglas McCloskey
Mads Nielsen
Christian Igel
Venue:  (2021)
Keywords: variational autoencoder
multimodal data
product-of-experts
semi-supervised learning

Abstract: Multimodal generative models should be able to learn a meaningful latent representation that enables a coherent joint generation of all modalities (e.g., images and text). Many applications also require the ability to accurately sample modalities conditioned on observations of a subset of the modalities. Often not all modalities may be observed for all training data points, so  semi-supervised learning should be possible.
In this study, we evaluate a family of product-of-experts (PoE) based variational autoencoders that have these desired properties. We include a novel PoE based architecture and training procedure. An empirical evaluation shows that the PoE based models can outperform an additive mixture-of-experts (MoE) approach.
Our experiments support the intuition that PoE models are more suited for a conjunctive combination of modalities while MoEs are more suited for a disjunctive fusion. 

Under review as a conference paper at ICLR 2021
MULTIMODAL VARIATIONAL AUTOENCODERS FOR SEMI-SUPERVISED LEARNING:
IN DEFENSE OF PRODUCT-OF-
Anonymous authors Paper under double-bling blind review
ABSTRACT
Multimodal generative models should able to learn a meaningful latent rep- resentation that enables a coherent joint generation of all modalities (e.g.. im- ages and text). Many applications also require the ability to accurately sample modalities conditioned
observations of subset of the modalities. Often not all modalities may be observed for all training data points, so semi-supervised learning should be possible. In this study, evaluate family of product-of- experts (PoE) based variational autoencoders that have these desired properties. We include novel PoE based architecture and training procedure. An empirical evaluation shows that the PoE based models can outperform an additive mixture- of-experts (MoE) approach. Our experiments support the intuition that PoE mod- els are more suited for conjunctive combination of modalities while MoEs are more suited for disjunctive fusion.
INTRODUCTION
Multimodal generative modelling is important because information about real-world objects typi- cally comes in different representations, modalities. The information provided by each modality may be erroneous and/or incomplete, and complete reconstruction of the full information can often only be achieved by combining several modalities. For example, in image- and video-guided trans- lation (Caglayan et al., 2019), additional visual context can potentially resolve ambiguities (e.g., noun genders) when translating written text.
In many applications, modalities may be missing for subset of the observed samples during train- ing and deployment. Often the deseription of an object in one modality easy to obtain, while annotating it with another modality is slow and expensive. Given two modalities, we call samples paired when both modalities are present, and unpaired one is missing. The simplest way to deal with paired and unpaired training examples is to discard the unpaired observations for learning. The smaller the share of paired samples, the more important becomes the ability to additionally learn from the unpaired data, referred as semi-supervised learning in this context (following the termi- nology from Wu & Goodman, (2019] Typically one would associate semi-supervised learning with learning form labelled and unlabelled data solve classification or regression tasks). Our goal is to provide model that can leverage the information contained in unpaired samples and to inves- tigate the capabilities of the model in situations of low levels of supervision, that is, when only few paired samples are available. While modality can be as low dimensional as a label, which can be handled by a variety of discriminative models (van Engelen Hoos 2020). we are interested in high dimensional modalities, for example an image and text caption.
Learning a representation of multimodal data that allows to generate high-quality samples requires the following: 1) deriving meaningful representation in joint latent space for each high dimen- sional modality and 2) bridging the representations of different modalities in a way that the relations between them are preserved. The latter means that do not want the modalities to be represented orthogonally in the latent space - ideally the latent space should encode the object's properties independent of the input modality. Variational autoencoders (Kingma & Welling, (014)) using product-of-exxe oxperts (PoE, Hinton 2002] Welling (007) approach for combining input modalities are promising approach for multimodal generative modelling having the desired properties, in partic-

---

Under review as a conference paper at ICLR 2021
ular the VAEVAE model developed by Wu & Goodman (2018) and novel model termed SVAE, which we present in this study. Both models can handle multiple high dimensional modalities, which may not all be observed at training time.
has been argued that a product-of-cxpent (PoE) approach is not well suited for multimodal gener- ative modelling using variational autoencoders (VAEs) in comparison additive mixture-of-experts (MoE). It has empirically been shown that the PoE-based MVAE (Wu & Goodman 2018) fails to properly model two high-dimensional modalities in contrast an (additive) MoE approach referred to as MMVAE, leading to the conclusion that "PoE factorisation does not appear to be practically suited for state-of-the art multimodal VAEs.
learning" (Shi et al. 2019). This study sets out test this conjecture for
The next section summarizes related work. Section Dintoduessss SVAE as an alternative PoE based VAE approach derived from axiomatic principles. Then we present our experimental evaluation of multimodal VAEs before we conclude.
BACKGROUND AND RELATED WORK
We consider multimodal generative modelling. We mainly restrict our considerations two modal- ities 2 E X1, 32 E X2, where one modality may be missing at a time. Extensions to more modal- ities are discussed in Experiments section and Appendix D To address the problem of generative cross-modal modeling, one modality can be generated from another modality 22 by simply us- ing independently trained generative models (a) + I2 and 2i + I1) or a composed but non- interchangeable representation (Wang et al. 2016 Sohn et al. 2015). However, the ultimate goal of multimodal representation learning find a meaningful joint latent code distribution bridging the two individual embeddings learned from 2 and 22 alone. This can be done by two-step procedure that models the individual representations first and then applies an additional learning step to link them (Tian & Engel 2019) [iilberer & Lapata 2014 Ngiam al. 2011). In contrast, focus approaches that learn individual and joint representations simultancousl Furthermore, our model should be able to learn in semi-supervised setting. Kingma et al. (2014) introduced two models suitable for the case when one modality is high dimensional (e.g., an image) and another is low dimensional (e.g., a label) while our main interest modalities of high complexity.
We consider models based on variational autoencoders ( (VAEs, Kingma & Welling, 2014, Rezende al. 2014). Standard VAEs learn latent representation z Z for a set of observed variables I E X by modelling a joint distribution p(æ, 2) p(z) 2). In the original VAE, the intractable posterior ((2 æ²) and conditional distribution p(x122 are approximated by neural networks trained by maximising the ELBO loss taking the form
L Fggt12)
p(x|==) p(<===
DKL (1(=)=) || NN I))
with respect to the parameters of the networks modelling (<([z) and p(z|z). Here D l|
denotes the Kullback-Leible divergence. Bi-modal VAEs that can handle a missing modality extend this approach by modelling q(=)x1, as well as 91 (=/221) and g2(2(x2), which replace the single ((2(z).. Multimodal VABS may differ in 1) the way they approximate <(>/01, (())=1,<2)) gi (2|z1) and
(=(2(22) by neural networks and/or 2) the structure of the loss function, see Figure [ Typicalll, there are no conceptual differences the decoding, and we model the decoding distributions in the same way for all methods considered in this study.
Suzuki et al. (2017) introduced a model termed JMVAE (Joint Multimodal VAE), which belongs to the class of approaches that can only learn from the paired training samples (what we refer to as the (fully) supervised setting). It approximates q(1[21, X22, (2(2(21) and (2(2((2)) with three corresponding neural networks and optimizes ELBO-type loss of the form
L = Fg( (lg (æ1|2) + logp2(и)|2)] D (g(=[[1, || ()),
DKa. (g(2/11,)) qi (2(œ1)) DKL DKL (((z))(/))))))))))))))))) x2) (=(222)) (2)
The last two terms imply that during learning the joint network output must be generated which requires paired samples.
The MVAE (Multimodal VAE) model (Wu & Goodman 2018) is the first multimodal VAE-based model allowing for missing modalities that does not require any additional network structures for
2

---

Under review as a conference paper at ICLR 2021
Zi
Z
Zu
PoE
PoE
PoE
PoE
9
WAEVAE
VAEVAE
SVAE
(proposed model)
Encoders
Decocers
Z
Zi
Pol
Gaussian prior
Z Latent space
PoE
Product of experts
MVAE
JMVAE
KL divergence
Figure 1: Schematic overview bi-modal VAEs using PoE and additional network structures that are capable of semi-supervised learning without requiring a two step learning procedure. VAEVAE (a) and (b) are by Wu & Goodman (2019), JMVAE by (uzuki al. (2017), MVAE is by Wu & Goodman (2018), and SVAE is our newly proposed model. Each triangle stands for an individual neural network, the colors indicate the two different modalities.
learning the joint latent code distribution. The joint posterior is modeled using a product-of-experts (PoE) as ((((Z1.M) = IIm 9m 9m (2(am)) For the missing modality qu [2]xx) = assumed. The model allows for semi-supervised learning while keeping the number of model parameters low.
The bridged model YYadvv et al. 2020) highlights the need for an additional network structure for approximating the joint latent code distribution. It attempts keep the advantages of the additional encoding networks. It reduces the number of model parameters by introducing the bridge encoder that consists of one fully connected layer which takes 2 and 22 latent code vectors generated from 21 and 12 and outputs the mean and the variance the joint latent code distribution.
The arguably most advanced multimodal VAE models is VAEVAE by Wu Goodman 2019 which we discuss in detail in the next section (see also Appendix C..
Shi al. (2019) proposed a MoE model termed MMVAE (Mixture-of-experrs Multimodal VAB). In MMVAE model the joint variational posterior for M modalities is approximated as (
The model utilizes loss function from the importance weighted autoencoder (IWAE, Burda et al.) 2016) that computes a tighter lower bound compared the VAE ELBO loss. The MoE rule formulation allows in principle train with a missing modal- ity by assuming a; = O, however, [Shi et al. (2019) do not highlight or evaluate this feature. There are benchmarks in the paper that compare M MVAE (Wu & Goodman 2018) and MMVAE, conclud- ing that MVAE often fails to learn the joint latent code distribution. Because of these results and those presented by Wu Goodman 2019), we did not include MVAE as benchmark model in our experiments.
Em Orn Om (2 (mm) where Om
VAEVAE AND SVAE
We developed approach alternative to VAEVAE. Both models are VAE based; 2) allow for interchangeable cross-mos model generation as well as a learning joint embedding; 3) allow for missing modalities training time; and 4) can be applied to similarly complex high dimensional

---

Under review as a conference paper at ICLR 2021
modalities. Next, will briefly present our model SVAE. Then highlight the differences to VAEVAE. Finally, we state a newly derived objective function for training the models. We consider two modalities and refer to Appendix for generalizations to more modalities.
SVAE Since both modalities might not be available for all the samples, it should be possible to marginalize each of them out of q(2(x1,222). While the individual encoding distributions ((2|211) and g(= (22) can be approximated by neural networks as in the standard VAE, we need to define meaningful approximation of the joint encoding distribution ((2/2)) (()221,22). In the newly proposed SVAE model, these distributions are defined as the following:
(22) 7 x2) 2 (2|20 (=((2))))
(<||<)) ) 2i (=|=)) (
(4)
(<(|22) = (=(2)=2) (z|=2)
(5)
((=) 1)
The model is derived from an axiomatic proof that is given in Appendix A The desired properties of the model were that 1) when modalities are observed the generating distribution for the latent code is Gaussian, 2) the modalities are independent given the latent code, 3) both experts cover the whole latent space with equal probabílities, and 4) the joint encoding distribution q(=|21, x2) is modelled by a PoE.
The distributions gi (2(zr). (2(222), (2(rx) and qi (2(222) are approximated by neural networks. In case both observations are available, q( (( is approximated by applying the product-of- experts rule with qı (2|22) and (2(2)x2) being the experts for each modality. In case of missing modality, equation 4 or is used. If, for example, 22 is missing, the (2(21) distribution takes over as "replacement" expert, modelling marginalization over 22.
SVAE vs. VAEVAE The VAEVAE model (Wu & Goodman 2019) is the most similar to ours. Wu
Goodman define two variants which can derived from the SVAE model in the following way.
9" (2(x22) = 1. Variant (b) is obtained from (a) by additionally using a separate network to model q(=|21, x2). Having a joint network g(2[21, x2)
Variant (a) can be derived by setting 9" (2|z1)
implements the most straightforward way of capturing the inteeddepenssncces of the two modalities. However, the joint network cannot be trained on unpaired data which can be relevant when the share supervised data gets smaller. Option (a) uses the product-of-experts rule model the joint distribution of the two modalities as well, but does not ensure that both experts cover the whole latent space (in contrast to SVAE, see equation A. in the appendix), which can lead individual latent code distributions diverging. Based on this consideration and the experimental results from Wu & Goodman (2019), we focused on benchmarking VAEVAE (b) and refer as simply VAEVAE in Section
SVAE resembles VAEVAE in the need for additional networks besides one encoder per each modal- ity and the structure of ELBO loss. It does, however, solve the problem of learning the joint embed- dings a way that allows to learn the parameters of approximated q(2 (21, ((2(21) X22) using all available samples, i.e., both paired and unpaired. If g( 2 1 x2) is approximated with the joint network that accepts concatenated inputs, as JMVAE and VAEVAE (b), the weights of q(>[211 x2) can only be updated for the paired share of samples. If q( 22, ) is approximated with a PoE of decoupled networks as in SVAE, the weights are updated for each sample whether paired or unpaired which the key differentiating feature of SVAE compared existing architectures.
4

---

Under review as a conference paper at ICLR 2021
A New Objective Function
When developing SVAE, we devised novel ELBO-type loss:
C -=E Pannoa
log -og_:ppopopllll)__ppppppoaaaal))
DKL. (9(2 (g(2)) (1, p(2(21))
Dx - p(z(22))
(o(slo)) |ogg1 (1)))))()2)))))))))) (1 [[=))
[ (ttalla)loog//260 [[gr) (x) [1=))
(7)
C1 = =E (11)
(o(=lan)loog/( -
(x(((())11) p()))
(8)
L2 E Puquin (1ra)
Dxc (2(2(22)
|| p(2))
(9)
Loomb =C + L1 + L2
(10)
Here Ppaired and Punpuired denote the distributions of the paired and unpaired training data, respectively. The loss function is derived in Appendix B The differences between this loss and the loss function used train VAEVAE by Wu & Goodman (2019) are highlighted in Appendix
4 EXPERIMENTS
7
7
We conducted experiments to compare state-of-the-art PoE based VAEs with the MoE approach MMVAE (Shi et al. 2019). We consid- ered VAEVAE (b) as proposed by Wu & Goodman (2019) and SVAE as described above. The two approaches differ both in the underlying model as well the objective function. For a better understanding of these differences, we also considered an algorithm referred to as VAEVAE*, which has the same model architecture as VAEVAE and the same loss function as SVAE The difference in the training proce- dure for VAEVAE and VAEVAΣ* is described in Appendix C Since the VAEVAE implementation was not publicly available at the time of writing, we used our own implementation of VAEVAE based on the PiXYZ library For details about the experiments we refer to Ap- pendix F The source code to reproduce the experiments can be found in the supplementary material.
1
9
9
9
3
0
Oracle
Oracle
Figure 2: MNIST-SVHN reconstruction for fully su- pervised VAEVAE.
For an unbiased evaluation, we considered the same test problems and performance metrics as Shi al. (2019). In addition, designed an experiment referred to as MNIST-Split that was supposed to be well-suited for PoE. In all experiments we kept the network architectures as similar as possible (see Appendix(F). For the new benchmark problem, modalities are similar in dimensionality as well as complexity and are providing missing information each other rather than duplicating it. The latter should favor PoE modelling. which suits an "AND" combination of the modalities, and not a MoE. modeling, which is more aligned with an "OR" combination.
constructed multi-modal dataset where the
We measured performance for different supervision levels for each dataset (e.g.. 10% supervision level means that 10% of the training set samples were paired and the remaining 90% were unpaired).
Image and image: MNIST-Split We created an image reconstruction dataset based on MNIST digits (LeCun al. 1998). The images were split horizontally into equal parts, either two or three depending on the experimental setting. These regions are considered as different input modalities. In the above notion of "AND" and "OR" tasks we implicitly assume an additional modality, which is the image label in this case. The fact that the correct digit can sometimes be guessed from only one part of the image makes the new MNIST-Split benchmark a mixture an "AND" and an "OR" task. This is in contrast to the MNIST-SVHN task described below, which can be regarded as an almost pure "OR" task.
TWO MODALITIES: MNIST-SPLIT. In the bi-modal version referred to MNIST-Split, the MNIST images were split in top and bottom halves of equal size, and the halves were then used
also evaluated SVAE*, our model with VAEVAR loss function, but never outperformed other models. fhtps//gitthb..cmmmmasss//yyyy
5

---

Under review as a conference paper at ICLR 2021
Supervision level
100%
0.1% %
Original image (a)
0
6
0
(b)
A
SVAE
Figure 3: MNIST-Split image reconstructions of top half and a bottom half given (a) the top half; (b) the bottom half of the original image.
as two modalities. We tested the quality of the image reconstruction given one both modalities by predicting the reconstructed image label with an independent oracle network, a ResNet-18 (He et al. 2016) trained on the original MNIST dataset. The evaluation metrics were joint coherence, synergy, and pross-coherence. For measuring joint coherence, 1000 latent space vectors the prior and both halves an image were then reconstructed with the corresponding decoding net- works. The concatenated halves yield the fully reconstructed image. Since the ground truth class labels do not exist for the randomly sampled latent vectors, we could only perform a qualitative evaluation, see Figure 3 Synergy was defined as the accuracy of the image reconstruction given both halves. Cross-coherence considered the reconstruction of the full image from one half and was defined as the fraction of class labels correctly predicted by the oracle network.
generated from
Table 1: Evaluation of the models trained the fully supervised datasets.
Accuracy (both)
Accuracy (top half)
Accuracy (bottom half)
MMVAE
0.539
0.221
0.283
SVAE
0,948
0,872
0,816
VAEVAE
0,956
0,887
0.830
VAEVAE
0,958
0.863
0.778
VIEWE
SMA
10
30
Sugervision Log Percent
Supervision Log Percent
Supervision Log Percent
Figure 4: MNIST-Split dataset. Accuracy of oracle network applied images reconstructed given (a) the full image (both halves) (b) the top half (c) the bottom half.
The quantitative results are shown in Table I and Figure All PoE architectures clearly outper- formed MMVAE even when trained on the low supervision levels. In this experiment, it is important that both experts agree on a class label. Thus, as expected, the multiplicative PoE. fits the task much better than the additive mixture. Utilizing the novel loss function (TO) gave the best results for very low supervision (SVAE and VAEVAE*).
THREE MODALITIES: MNIST-SPLIT-3. We compared simple generalization of the SVAE model to more than two modalities with the canonical extension of the VAEVAE model (both de- fined in Appendix D the MNIST-Split-3 data, the 3-modal version of MNIST-Split task. Figure

---

Under review as a conference paper at ICLR 2021
Encaders
N
X1 AB
12, Xa
XL Xb
Z
2
Decoders
Zi
Zu
Zi3
X1
X2
Z125
Z123
N
Gaussian prior
2
Latent spece
KL divergence
q1
4/2
4/3
Q2
9²>
VAEVAE
SVAE
Figure 5: The SVAE and VAEVAE network architectures for 3-modalities case. The number of parameters is kn for SVAE and kn2n-1 for VAEVAE, where n is the number of modalities and is the number of parameters in one encoding network.
Rull
Middle
Bctton
32
27
30+
Supervision Log-Prrcent
Supersision Lag-Percent
Suppervisionn Lag-Percent
Supervision Log- Percent
Figure 6: MNIST-Split-3 dataset, reproducing the logic of MNIST-Split for the images splitted in three parts.
shows that SVAE performed better when looking at the individual modalities reconstructions. While the number of parameters in the bi-modal case the same for SVAE and VAEVAE, grows expo- nentially for VAEVAR and stays in order of n² for SVAE where is the number of modalities, see Figure 5 and Appendix [ for details.
Image and image: MNIST-SVHN The first dataset considered by Shi al. (2019) is constructed by pairing MNIST and SVHN (Netzer et al., 2011) images showing the same digit. This dataset shares some properties with MNIST-Split, but the relation between the two modalities different: the digit class is derived from a concatenation of two modalities in MNIST-Spli while in MNIST- SVHN it could be derived from any modality alone, which corresponds to "AND" combination of the modalities and favors the MoE. architecture. As before, oracle networks are trained to predict the digit classes of MNIST and SVHN images. Joint coherence was again computed based on 1000 latent space vectors generated from the prior. Both images were then reconstructed with the corresponding decoding networks. A reconstruction was considered correct the predicted digit classes of MNIST and SVHN were the same. Cross-coherence
measured above.
Figure 2 shows examples of paired image reconstructions from the randomly sampled latent space of the fully supervised VAEVAE model. The digit next the each reconstruction shows the digit class prediction for this image. The quantitative results in Figure /ssow that all three PoE based models reached similar joint coherence as MMVAE, VAEVAE scored even higher. The cross-coherence
7

---

Under review as a conference paper at ICLR 2021
gint coherence
Cross coherence (M
Cross coherence (5 M)
VAEVAE VAEVAE+
0.8
MMWAE, supervised
NVRE: supervised
0.2
0.0
10
10
10
10a
10
10
10"
10
10
10"
10"
10'
Supervision Log Percent
Supervision Log Percent
Supery sion Log Percent
Figure 7: Performance on MNIST-SVHN for different supervision levels. (a) Joint coherence, share of generated images with the same digit class; (b) Cross-coherence, accuracy of SVHN recon- structions given MNIST; (c) Cross-coherence, accuracy of MNIST reconstructions given SVHN.
results were best for MMVAE, but the three PoE based models performed considerably better than the MVAE baseline reported by Shi et al. (2019).
Image and text: CUB-Captions The second benchmark considered by Shi et al. (019)) is the CUB Images-Captions dataset (Wah et al. (011)) containing photos of birds and their textual de- scriptions. Here the modalities are of different nature but similar in dimensionality and information content. We used the source code by Shi et al.) to compute the same evaluation metrics in the MMVAE study. Canonical correlation analysis (CCA) was used for estimating joint and cross- coherences of images and text (Massiceti al. 2018). The projection matrices W,, for images and Wy, for captions were pre-computed using the training set of CUB Images-Captions and are avail- able as part of the source code. Given new image-caption pair ã, ÿ. we computed the correlation o(3) a(?) where ((k) W avg(W? k).
between the two by corr(ā, 5) [|
We employed the same image generation procedure in the MMVAE study. Instead of creating the images directly, we generated 2048-d feature vectors using a pre-trained ResNet-101. 101. In order to find the resulting image, nearest neighbours lookup with Euclidean distance was performed. A CNN encoder and decoder was used for the (see Table F.5 and Table F.6) Prior to computing the correlations, the captions were converted to 300-d vectors using FastText (Bojanowski al. 2017).
As in the experiment before, we used the same network architectures and hyperparameters as Shi al. (2019). We sampled 1000 latent space vectors from the prior distribution. Images and captions were then reconstructed with the decoding networks. The joint coherence was then computed as the CCA for the resulting image and caption averaged over the 1000 samples. Cross-coherence was computed from caption image and vice versa using the CCA averaged over the whole test set.
As can be seen in Figure g VAEVAE showed the best performance among all models. With full supervision the VAEVAE model outperformed MMVAE in all three metrics. The cross-coherence of the three PoE models was higher or equal to MMVAE except for very low supervision levels. All three PoE based models were consistently better than MVAE.
DISCUSSION AND CONCLUSIONS
We studied bi-modal variational autoencoders (VAEs) based (PoE)architecture, in particular VAEVAE as proposed by Wu & Goodman (2019) and a new model SVAE, which we derived in axiomatic way, and represents generalization of the VAE- VAE architecture. The models learn representations that allow coherent sampling of the modalities and accurate sampling of one modality given the other. They work well in the semi-supervised setting, that is. not all modalities need to be always observed during training. It has been argued that the cexture-of-experre (MoE) approach MMVAE preferable PoE for multimodal VAEs (Shi et al. 2019). in particular in the fully supervised setting (i.e., when all data are paired).
product-of-experts
8

---

Under review as a conference paper at ICLR 2021
jont coherence
Cross coherence
MEVS Cross coher ence 15
0.3
025
022
MARE
015
-010
Sepervision
Supervision Log
Supervision Log
Figure 8: CUB Images-Caption dataset. Performance metrics for different supervision levels. (a) Joint coherence, the correlation between images and labels reconstructed from the randomly sam- pled latent vectors; (b) coherec, the correlation of the reconstructed caption given the image; (c) Cross-coherence, the correlation of the reconstructed image given the caption.
SVAE
VAEVAE
SVAE
VAEVAE
This bird has wings black has a red belly
This bird has black feathers with bright
The bird has along black that and curved
llarge bird has white and and blackerenge bill
Original image
Reconstruction from the image
Original image
Reconstruction from the image
SVAE
VAEVAE
SVAE
VAEVAE
This bird black with white and has long. pointy
white seagull with brown top standing on a wooden clock
This
A small bird with white and and brown wings
This large bird bird has a and feathers its wings
white and has long. pointy beak
white and has long. pointy beak
Original caption
Reconstruction from the caption
Original caption Reconstruction from the caption
Figure 9: Examples of image and caption reconstructions given one modality input for SVAE and VAEVAE. Given that the caption can be broad (e.g., "this bird is black and white and has long pointy beak" in the example), it can fit many different images. In this case, the image from the caption reconstruction tends to better fit the description than the original image. The same goes for images: one of the reconstructed images has a bird with a red belly which got reflected in the generated caption even though not part of the original caption.
This conjecture was based on a comparison with the MVAE model (Wu & Goodman 2018) but is refuted by our experiments showing that VAEVAE and our newly proposed SVAE can outperform MMVAE on experiments conducted by (Shi et al. (2019). Intuitively, MoEs are more the information provided by the different modalities, while PoEs are more tailored to towards "AND" (multiplicative) combination. This demonstrated by our experiments on halved digit images, where a conjunctive combination helpful and the PoE models perform much better than MMVAE. We also expand SVAE and VAEVAE to 3-modal case and show that SVAE demonstrates better performance on individual modalities reconstructions while having less parameters than VAEVAE.
tailored towards an "OR" (additive) combination

---

Under review as a conference paper at ICLR 2021
REFERENCES
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information, Transactions of the Association for Computational Linguistics, 5:135-146, 2017.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders, In Inter- national Conference on Learning Representations (ICLR), 2016.
Ozan Caglayan, Pranava Madhyastha, Lucia Specia, and Loie Barrault. Probing the need for vi- sual context in multimodal machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT), volume 1, pp. 1159-41770 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016.
Geoffrey E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8):1771-1800, 2002.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In International Confer- ence on Learning Representations (ICLR), 2014.
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in Neural Information Processing Systems (NeurIPS). pp. 3581-3589, 2014.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-232) 1998.
Daniela Massiceti, Puneet K. Dokania, N. Siddharth, and Philip H. S. Torr. Visual dialogue with- out vision or dialogue. In NeurIPS Workshop on Critiquing and Correcting Trends in Machine Learming. 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop Deep Learning and Unsupervised Feature Learning 201 1, 2011.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y. Ng. Mul- timodal deep learning. In Proceedings of the 28th International Conference on International Conference on Machine Learning (ICML), pp. 689-696, 2011.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference deep generative models. In Proceedings of the 31 International Confer- ence on Machine Learning (ICML), volume 32(2) of Proceedings of Machine Learning Research, pp. 1278 12866 PMLR, 2014.
Yuge Shi, Siddharth N, Brooks Paige, and Philip Torr. Variational mixture- ofxtuure-of-expetss autoencoders for multi-modal deep generative models. Advances in Neural Information Processing Systems (NeurIPS), pp. 15718 177891 2019.
Carina Silberer and Mirella Lapata. Learning grounded meaning representations with autoencoders. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), volume 1, pp. 721-732. Association for Computational Linguistics, 2014.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems (NeurIPS), pp. 3483-3491, 2015.
10

---

Under review as a conference paper at ICLR 2021
Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Joint Multimodal Learning with Deep Generative Models. In International Conference on Learning Representations Workshop (ICLR) Workshop Track, 2017.
Yingtao Tian and Jesse Engel. Latent translation: Crossing modalities by bridging generative mod- els, 2019, MEXiv:1902.0826
Jesper van Engelen and Holger H. Hoos. survey on semi-supervised learning. Machine Learn- ing. 109(2):3733444 2020.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech- UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.
Weiran Wang, Xinchen Yan, Honglak Lee, and Karen Livescu. Deep variational canonical correla- tion analysis, 2016. arXiv:1610.03454.
Max Welling. Product of experts. Scholarpedia, 2(10):3879, 2007.
Mike Wu and Noah Goodman. Multimodal generative models for scalable weakly-supervised learn- ing. In Advances Neural Information Processing Systems 31, pp. 5575-5585, 2018.
Mike Wu and Noah Goodman. Multimodal generative models for compositional representation learning, 2019. 191.05555
Ravindra Nath Yadav, Ashish Sardana, Vinay P. Namboodiri. and Rajesh M. Hegde. Bridged vari- ational autoencoders for joint modeling of images and attributes. 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1468-1476, 2020.
A DERIVATION OF THE MODEL ARCHITECTURE
We define our model in an axiomatic way, requiring the following properties:
When no modalities are observed, the generating distribution for latent code Gaus- sian:
<(=) = =(=) = (O,
(A.11)
This property is well known from VAEs and allows easy sampling.
2. The two modalities are independent given the latent code, so the decoder distribution is: M(11ι421=
(a (z))2
(A.12)
The second property formalizes our goal that the latent representation contains all relevant information from all modalities.
The joint distribution p(=[0) = p(2|01, x22) is given by
p(2|21, #2) = p(=)p1 (21,
(2)))
(2)
A.12
(la)p
(A.13)
3. Both experts cover the whole latent space with equal probabilities:
(=) / gi (z(ar) (p())))( /
(2(2|12) g2(z)
(A.14)
4. The joint encoding distribution q(2(0r) = g(2 æ1, 42) is assumed to be given by the product-
of-experts nule (Hinton 2002 Welling 2007
(=(001,22) Z Z( x2)
(2)
(A.15)
(22 de'. The modelling by a producctoffexperee in equa- tion A.15 is simplification of equation A.13 to make the model tractable.
with Z [2) x2)) ) (2'(0z (2)((')))))))))))))))))))))))))

---

Under review as a conference paper at ICLR 2021
Given equation A.15 and equation A.14 obtain ((2) - (222))((z)d=
A.15
1
21 (al (A.16)
x2)
Let us define
(2(2)))) ((x xz))
of (
(A.17)
and write
((2 [ (1(2211)(())) / (xx1
/ p(an ptca)allleeDDealalaaaa (A.18)
So the proposal distributions are:
(=||0,,02) = 2( x22)
a1 (22(11222(=
(A.19)
(((|=1) = (2(21)
(A.20)
(2((12) ( (2(o2)
(A.21)
((=) = ())
(A.22)
B DERIVATION OF THE LOSS FUNCTION
In the following, derive the ELBO-type loss use for training, equation 7 Let consider the optimization
1
Eppo 22)) (logglpa,,,i_ p(x1, x2)] Pu (or 22) [logp(m (22) log log ) tog logp(z)))
2
2 E
Emnna .x) (log pl p [x)) +
E Pmp p1o
2
F (x2) p( + Ponn(en (B.23)
We can now proceed by finding lower- bounds for each term. For the last two terms log p(x;) we can use the standard ELBO given in equation This gives the terms
L: (x))((e)) (g(xter. (lope:))llpppp:ppp
(x((((()z)) || p()))
(B.24)
Next, we will derive log p(z) (x2). This we can do in terms of a conditional VAE (Sohn et al, 2015), where we condition all terms 22 (or 21 if model log pogg((2aan (x))) So the model derive the log-likelihood for is p( ((_]|<²)) = (((rl==)p(= (=)p((()))))d)) (2) where p(z (222) is now our prior. By model assumption we further have p( 22, 2) p(z (2))(())))))(((((()))))))) and therefore p(z1 [x2, 2) p(z1 Thus we arrive at the ELBO losses
L12 = E (x)) .mn)
(g [log (ogg (x1
(g(2/// p(2[2>))
(B.25)
and
L21 Epona(ar.2)) [ [
P2
DRL (x( p(2|=1))]
(B.26)
We now insert the terms in equation B.23 and arrive at:
E .r2) [og.((1,, S C12 + Lei L1 Ce
= Epon
E Eg( plog (21|²)] D p(2[x2))
+ F (e 122)
( log (22 (ka(((=((0, 22) p(2[=1)]]
+ E Fp))(o))
E E
flog (21[2)] - Dka(g(=/m)) [l
E (22)
(log 1atp://llll)---Do((laaaaala
(g(((() [[)))
12

---

Under review as a conference paper at ICLR 2021
The first two terms together give
Ppn 22 (x11l2) + [og22(22|=)]]
(B.27)
We do not know the conditional prior p(z(z,). By definition of the VAE, we are allowed to optimize the prior, therefore we can parameterize it and optimize it. However, we know that in an optimal model p(=[zx) x ((=/0;) and it might be possible to prove that if p(=[2;) is learnt in the same model-class as ((2)x;) we can find that the optimum indeed p(z[2+) g(2[x)). Inserting this choice into the equation gives the end-result.
C TRAINING PROCEDURE FOR SVAE AND VAEVAE* *
Algorithm 1: Training procedure for SVAE and VAEVAE* In bold terms that are different
from Wu & Goodman (2019)
Input: Supervised example (I1, Unsupervised example 2/j, Unsupervised example z/2
z' = 222 22)
Em Q1 (2(21)
Zmz = (2(2(x22
d1 = DKL(q(>') Ø1, 22
(Zn, x1)) + DKL (q1 (Zz, x1 [pp())(()))
d2 DKL(g(2' x1, 22
12 Raz x2)) + DKL (42 (Fxz x2) llp)))
= log pı (x1
2)
+ P2g22(22) 2) + log P1 (I1 (xs) + log Pa (x2) + d1 + d2
Cx1 log P1 [x]]
iz
DKL (91 (2z, x [l(()))
D.D. = log P2 a
Rzz
+ DKL. (92 2z, 22 (22))
Leomb L' L Laz
D SVAE AND VAEVAE FOR MORE THAN TWO MODALITIES
In the following, we formalize the VAEVAF model for three modalities and present a na've extension of the SVAE model more than two modalities.
In the canonical extension of VAEVAE to three modalities, the three- and two-modal relations are captured by the corresponding networks q2|/=1, 22, rg), q(=|=;,,;) and q(>[2;) for i, j {1, 3),, see Figure In the general n-modal case, the model has 2" networks. For 3, the loss function reads:
1,2 = -E E ma.
+ log log22())))) + log (2(a)2))
DDx [11 22, 23) || (2)(21,22))
DxL((( [21, 22, m0)
(22(22, 23))
Dxi(g(z(an) (2220 22)
|| 23))
(D.28)
Cgj Fymmalli w2)
(P (log (x,12) log (x,
DDx( x3)) (([[21)) DKL (1( || (((())))
(g1l((=2 (==(==)) DRI DRa (1(2/1),)))) (=))
(D.29)
C. Eponne(((;)) E [log p. (x,|22] (g((2())))) (x))
(D.30)
3
Lomb = -L,,23 +
Σ
hi +
Σ
C
(D.31)
In this study, considered simplifying extension of SVAE to modalities using n² networks of (z|ay)) for fori,j i,j E (l,-.., n). For the 3-modal case depicted in Figure the PoE relations between
13

---

Under review as a conference paper at ICLR 2021
the modalities are defined in the following way:
(21 = Z 22, 23)
(D.32)
J. k E (
(D.33)
i
(=(1)11))))
of (2
(D.34)
Z(ri 2z)
(x(1)1,)))) = Z( 2z)
(2
(D.35)
(<11x;)) - (=|xx) (zl <(=) 10,
(D.36)
(D.37)
The corresponding SVAE loss function has additional terms due to the fact that the relations between
pairs of modalities need to be captured with PoE rules q' (2|æ;, xz)) and q' (2(22,, in SVAE, while there is only single network g({(21,T5) in VAEVAE. The loss functions equation D.28 equation D.31 above are modified in a way that [(((=/a, m3)) = (2(wi,,)))) + (( (z) (z/0a,, Tt)) for any function f.
This extension of the bi-modal case assumes that p(z,, Ty (24) = p( ( ((,y (2x) for i, l E i1, # + k, which implies that Ty, Iyy and xg are independent of each other.
E QUALITATIVE EXAMPLES
E.I SVAE
See Figure E.10
E.2 VAEVAE
See Figure E.11
F TRAINING DETAILS AND ENCODER/DECODER ARCHITECTURES
The encoder and decoder architectures for each experiment and modality are listed below. To imple- ment joint encoding network (VAEVAE architecture), an fully connected layer followed by ReLU added to the encoding architecture for each modality. Another fully connected layer accepts the concatenated features from the two modalities as an input and outputs the latent space parameters. Adam optimiser is used for learning in all the models (Kingma & Ba 2015).
F.1 MNIST-SPLIT
The models are trained for 200 epochs with the learning rate 10 The best epoch chosen by the highest accuracy of the reconstruction from the top half evaluated the validation set. We used a latent space dimensionality of T. = 64. The network architectures are described in Table
Encoder
Decoder
Input 3=32a332
Input R
4x4 conv. 64 stride 2 pad I & ReLU
FC. L & ReLU
4x4 conv. 128 stride pad & ReLU
FC. 512 & ReLU
4x4 conv. 256 stride 2 pad & ReLU
FC. 112 & ReLU
FC. 786 ReLU
4x4 upconv. 56 stride pad 0 & ReLU
FC. L, FC. L
4x4 upconv. 28 stride 2 pad I & ReLU
Table F.2: Network architectures for MNIST-Split for each image half.
14

---

Under review as a conference paper at ICLR 2021
SVAE
100%
0.1%
MNIST-Split
MNIST-SVHN
MNIST-Split
MNIST-SVHN
Orginallmaa From-hh thp Fomm the botton From
Orignaliage
Figure E. A. MNIST-Split image reconstructions of top half and a bottom half given (a) the top half; (b) the bottom half of the original image (c) both halves. B. Side-by-side MNIST-SVHN reconstruction from randomly sampled latent space, with oracle predictions of a digit class. The joint coherence from the Figure 7 share of classes predicted the same. The examples are generated by SVAE for the supervision levels 100% and 0.1% 1%
F.2 MNIST-SVHN
The models are trained for epochs with the learning rate 10-3 VAEVAE and VAEVAE* and 10-4 for SVAF. The best epoch is chosen by the highest joint coherence evaluated the validation set. We used a latent space dimensionality of L = 20. The network architectures are summarized in Table F.4 and Table F.3 for the MNIST and SVHN modality, respectively.
Encoder
Decoder
Input E IR 111288228
Input E RR4 FC. 400 ReLU
FC. 400 & ReLU FC. L, FC. L
FC. x 28 x 28 Sigmoid
Table F.3: Network architectures for MNIST-S SVHN: MNIST.
15

---

Under review as a conference paper at ICLR 2021
VAEVAE
100%
0.1%
MNIST-Split
MNIST-SVHN
MNIST-Split
MNIST-SVHN
Oriiinal
7
Original
From
Foom the bottom
Frombooh
Figure E.11: A. MNIST-Split image reconstructions of top half and a bottom half given (a) the top half; (b) the bottom half of the original image (c) both halves. B. Side-by-side MNIST-SVHN reconstruction from randomly sampled latent space, with oracle predictions of a digit class. The joint coherence from the Figure by VAEVAE for the supervision levels 100% and 0.1%
share of classes predicted the same. The examples are generated
F.3 CUB-CAPTIONS
The models are trained for 200 epochs with the learning rate 10-4 The best epoch is chosen by the highest joint coherence evaluated on the validation set. We used latent space dimensionality of 64. The network architectures are described Table FS and F.ble|F.6 for the text and image modality, respectively.
F.4 MNIST-SPLII-
The models are trained for 50 epochs with the learning rate 2 The best epoch is chosen by the highest accuracy of the reconstruction from the top half evaluated the validation set. We used a latent space dimensionality of T. = 64. The network architectures are described in TableIF7
16

---

Under review as a conference paper at ICLR 2021
Encoder
Input RR3332232
4x4 conv. 32 stride 2 pad & ReI.U
4x4 conv. 64 stride 2 pad & ReLU
4x4 conv. 128 stride pad & ReLU
4x4 conv. stride pad 0, 4x4 conv. stride pad 0
Decoder
Input E IR
4x4 upconv. 128 stride pad & ReLU
4x4 upconv. 64 stride 2 pad & ReLU
4x4 upconv. 32 stride 2 pad & ReLU
4x4 upconv. 3 stride 2 pad & Sigmoid
Table F4:: Network architectures for MNIST-SVHN: SVHN.
Encoder
Input 1290
Word Emb. 256
4x4 conv. 32 stride 2 pad & BatchNorm2d & ReLU
4x4 conv. 64 stride 2 pad & BatchNorm2d & ReLU
4x4 conv. 128 stride pad & BatchNorm2d & ReLU
1x4 conv. 256 stride 1x2 pad 0x & BatchNorm2d & ReLU
1x4 conv. 512 stride 1x2 pad 0x1 & BatchNorm2d & ReLU
4x4 conv. L stride I pad 0, 4x4 conv. L stride pad 0
Decoder
Input E R-
4x4 upconv. 512 stride pad 0 & ReLU
1x4 upconv. 256 stride 1x2 pad 0x1 & BatchNorm2d & ReLU
1x4 upconv. 128 stride Ix2 pad 0x & BatchNorm2d & ReLU
4x4 upconv. 64 stride pad & BatchNorm2d & ReLU
4x4 upconv. 32 stride pad & BatchNorm2d & ReLU
4x4 upconv. stride 2 pad & ReLU
Word Emb.T 1590
Table F.5: Network architectures for CUB- CUB-Captions language processing.
Encoder
Decoder
Input E 2048
Input R FC. 256 ELU
FC. 1024 ELU
FC. 512 ELU
FC. 512 ELU
FC. 256 ELU
FC. 1024 ELU
FC. L,, FC. L.
FC. 2048
Table F.6: Network architectures for CUB-Captions image processing.
17

---

Under review as a conference paper at ICLR 2021
Encoder
Decoder
Input E Sa32u332
Input R
4x4 conv. 64 stride 2 pad & ReLU
FC. L & ReLU
4x4 conv. 128 stride pad & ReLU
FC. 512 & ReLU
4x4 conv, 256 stride pad & ReLU
FC. 112 & ReLU
FC. 786 & ReLU
4x4 upconv. 56 stride 1 pad 0 & ReLU
FC. L,, FC.
2x4 upconv. 28 stride 2 pad I & ReLU
Table F.7: Network architectures for MNIST-Split-Three for each image part.
18


Reviews:
[{"review_id": "m4PPGFFKOz", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper discusses and evaluates different generative models for multimodal data. Specifically, the authors are interested in comparing product-of-experts (PoE) to mixture-of-experts (MoE) in VAEs as ways to handle multimodality.\n\nThey also propose a novel model (SVAE) built on the PoE approach that, compared to previous models, aims to better handle missing modalities. In particular, they introduce additional networks that estimate the marginal distributions of the latent representations of the missing modalities given the observed ones.\n\nThey then evaluate multiple models on three tasks that highlight the difference of behavior in PoE and MoE-based models, concluding that PoEs are useful to model \"AND\" relationships in a multimodal setting.\n\n\n################################################\n\nStrong points:\n\n-The paper is easy to follow, clearly presents the context and the different approaches. The proposed model is also elegantly designed and presented, with differences with previous methods highlighted.\n\n-The authors seem to have provided most of the information needed for reproducibility.\n\n\nWeaknesses:\n\n-While SVAE is useful in the context of the paper as it uses explicit PoEs, the experiments don't show conclusive differences in performances with VAEVAE and VAEVAE* variants, reducing the potential impact of the novel elements of the paper. Actually, VAEVAE* (that is very close to the prior existing VAEVAE) obtains better or comparable performances except on the MNIST-split task with 1% or less paired data.\nAlso, uncertainty bars in the plots would be very useful, especially since there seem to be very large variations in what should theoretically be mostly monotonous curves (Figure 4 and Figure 6 left).\n\n-The MNIST-split results are used to support the idea that PoEs are better-suited to model \"AND\" relationships (both modalities carry complementary information and are needed for the task). However, it seems that a system (composed of VAEVAE and the oracle) that takes as input only the top part is able to predict the correct class with an accuracy of 0.887. That tends to indicate that the evaluated task is fundamentally an \"OR\" task, and therefore ill-suited to evaluate the adequacy of the method to capture \"AND\" relationships. Can the authors comment on this potential issue?\n\n-The paper is articulated around proving that PoE models can obtain good performances. Since VAEVAE already performs well and that the opposite view is only supported by a single publication, the importance of this particular contribution is questionable.\n\n\n################################################\n\nScore motivation:\n\nMainly, I believe the shown results aren't conclusive enough. They can't support the proposed model, nor the other insights of the paper.\n\n\n################################################\n\nOther question:\n\nCan the authors comment on why MMVAE is terrible at auto-encoding an MNIST image (0.539 accuracy in table 1)? It seems very low.\n\n\n\n################################################\n\n################################################\n\nPost-Rebuttal Update:\n\nI'd like to thank the authors for their updates, the additional experiments are especially welcome. After taking into consideration the responses and the new evidence, I believe the concerns I raised still stand. Therefore, I keep the previous rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "d3r1AUDQlDJ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\nThis paper proposed SVAE as a more general form for the previous VAEVAE model.\n\nCons:\n1). In the subsection \"Image and text: CUB-caption\", the definition of \\phi(\\tilt{k}) is a little bit confusing. Is there a typo (\\tilt{k} == k), or what is k, a constant?\n2). From all the experiments listed in the paper, we can see that VAEVAE performs best. And the content is lack of analysis why VAEVAE is better than SVAE in practice. From the subsection \"SVAE vs VAEVAE\" we know that SVAE is a general form of VAEVAE, but it becomes less important since the general form performs worse than a special case in practice. In other words, the importance of the works is not that convinicing. At least, as a reader, we expect to see in some cases, SVAE is better than VAEVAE.\n3) In equation (10), will weighted sum be a better choice?", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZVKCjF2BlQ5", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a variant of multimodal VAE model. It also argues in favour of product-of-experts approaches vs. additive mixture-of-experts ones.\nThe paper clearly frames the contribution within the relevant literature, the introduction is well written and the paper is well structured. Variants of multimodal VAE are also introduced (although acronyms are not made explicit, which would make the exposition clearer), and the new derivation is presented and explained.\nClaims are supported by experimental results, that use MNIST, SVHN, CUB-Captions. A final discussion summarises the claimed contribution and advantages of the proposed approach.\n\nComments:\n- Please make acronyms explicit early on to make the exposition clearer\n- Figure 1 is a nice visualisation, very helpful\n- Does the proposed approach generalise easily to more than two modalities?\n- How well do you expect your approach to work on modalities presenting considerably different dimensionality (also in comparison with other approaches)?\n- Quantitative results show comparable scorse between SVAE and VAEVAE variants. Where/why would SVAE be most beneficial/advantageous? \n- Which applications would see MoE approaches outperform PoE? I think this point is important for completing the analysis and comparison of the two approaches. Is there a case where PoE and MoE perform comparably, and why would that happen?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "AHOVYBgifRy", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Update\nI thank the authors for their comments and answers. While they agree on some of the concerns I raised, others are still left open.\nI believe they could be addressed in new major revision of the paper and I encourage authors to do so.\n\n# Summary\nThis paper qualifies as a discussion around architectural choices when using variational-autoencoders (VAEs) for multi-modal learning.\nThe main claim is that architectures supporting the mixture-of-experts (MoEs) paradigm favor benchmarks where modalities appear in an 'OR'-relationship, while those implementing products-of-experts (PoEs) favor 'AND'-relationships.\nTo show this point and in 'defense' of PoE-VAEs, the paper introduces a missing-value imputation experiment over MNIST images to evaluate the second aspect.\nAs another contribution, the authors introduce the SVAE as a variation of a sota PoE-VAE, the VAEVAE, for bi-modal learning. Specifically, they introduce a variation that employs auxiliary encoder networks for each modality and derive accordingly a new composite ELBO loss to train them.\n%\nThe point raised by author is reasonable, but the execution of the experimental section and little relevance of the introduced SVAE limit the value of the paper in its current state. Detailed comments follow.\n\n\n# Presentation\nThe paper is generally understandable and well-written.\nThe introduction might benefit a rewriting as to gently introduce the larger scope of multimodal learning and then focusing on the more recent advancements, such as PoE-VAEs and MoE-VAEs.\nFor example, the first lines of the first paragraph directly jump to PoEs without providing the reader enough background.\n\nI suggest authors to clarify that the scenario they are tackling is that of generative modeling in presence of missing data on the _inputs_ (as a modality missing is a special case of missing not at random) more than referring to it as semi-supervised learning.\nI recognize that the authors follow a recent trend in the VAE-literature, but classically, semi-supervised has been referred to as the case when missing values are on the _output_, in a clear discriminative setting. \n\nSome typos are present, e.g., \"which corresponds to ”AND” combination of the modalities and favors the MoE architecture\" (should be \"OR\"). Clearly, they can be easily fixed by an additional proofreading pass.\n\n# Contributions\nOn the one hand, the discussion of the pros and cons of PoE-VAEs and MoE-VAEs boils down to one conjecture ('AND' versus 'OR' modality fusion) that is shallowly tested in the experimental section.\nWhich can be greatly strengthened, see comments below.\n\nOn the other hand, the introduction of the SVAE architecture, while potentially appealing, seems a minor contribution after seeing that VAEVAE and MMVAE generally outperform it in the graphs and tables in the experimental section (with the exception of the first experiment, joint modalities and one single percentage of low supervision).\nI would advice authors to explicitly say what is the benefit (maybe didactic?) of introducing the SVAE as a new model.\n\nFurthermore, a limitation of a certain regard concerns dealing with bi-modal data only in the text and in the experiments. \nThis makes unclear what is the price to pay to scale these architectures to truly-multimodal data.\nFor examples, Eqs. 7-1 suggest that the new composite ELBO can be extended to include uni-modal ELBO terms for each modality. However, one ordering over modalities for conditioning (according to Appendix B) shall be chosen, and it is not clear how this can influence learning and inference (in a similar fashion variable ordering influences autoregressive models).\nArchitecture-wise, it is not self-evident how many additional encoder components are needed for more than two modalities. If more than one per modality, then the challenges should be discussed in depth.\n\n# Experiments\nAs already stated, experiments are limited to bi-modal data only.\nOne additional downside of the experiments is that only coherence inter-modalities is measured as a metric. Sample/modality quality is not discussed, not even reported in a qualitative way (for the exception for some samples for the CUB dataset and some MNIST image reconstructions only for SVAE in Fig.3).\nI suggest authors to report the FID scores (or any other suitable variant like KID or precision-recall curves) of the joint and single modalities for the generated images to assess their quality. This is a fundamental aspect as modalities can be coherent but very far from the true data distribution or still not exactly close to the reconstruction, which is just a mode of the whole distribution.\nAlong this direction, one shall evaluate conditional sampling and not only reconstructions to see if the VAE have collapsed to pointwise densities.\n\nLastly, the CUB experiment provides some empirical evidence that is hard to evaluate or pose in the context of generative modelling.\nIn order to follow the MMVAE paper, the authors are decoding images not in pixel space, but in the latent space of a  ResNet-101. Then the showed images are the nearest neighbours in the training data.\nThis is the opposite of the generative modelling paradigm, and misleading: equivalently accurate and good-looking final images could come from a model memorizing the training set.\nI advice authors to evaluate the quality of generated samples and reconstructions in the pixel space (with the metrics discussed above), alternatively, to introduce a proper decoder for the ResNet-101 embedding or not to include the above experiment at all.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers found this to be an interesting and clearly-written paper, but broadly agreed that it is not yet ready for acceptance. In particular, multiple reviewers felt that the experiments don't show clear benefits of the proposed SVAE approach when compared to the VAEVAE and other baselines; nor do they sufficiently back up the central claim regarding relative benefit of PoE vs MoE for either "AND" or "OR" relations. Hopefully the comments and suggestions from the reviewers, particularly regarding framing and experimental validation, will help in revising the paper.
Decision: Reject