Year: 2021

Abstract:
Multimodal learning for generative models often refers to the learning of abstract concepts from the commonality of information in multiple modalities, such as vision and language. While it has proven effective for learning generalisable representations, the training of such models often requires a large amount of “related” multimodal data that shares commonality, which can be expensive to come by. To mitigate this, we develop a novel contrastive framework for generative model learning, allowing us to train the model not just by the commonality between modalities, but by the distinction between “related” and “unrelated” multimodal data. We show in experiments that our method enables data-efficient multimodal learning on challenging datasets for various multimodal variational autoencoder (VAE) models. We also show that under our proposed framework, the generative model can accurately identify related samples from unrelated ones, making it possible to make use of the plentiful unlabeled, unpaired multimodal data.

text: 1 INTRODUCTION
To comprehensively describe concepts in the real world, humans collect multiple perceptual signals of the same object such as image, sound, text and video. We refer to each of these media as a modality, and a collection of different media featuring the same underlying concept is characterised as multimodal data. Learning from multiple modalities has been shown to yield more generalisable representations (Zhang et al., 2020; Guo et al., 2019; Yildirim, 2014), as different modalities are often complimentary in content while overlapping for common abstract concept.
Despite the motivation, it is worth noting that the multimodal framework
is not exactly data-efficient—constructing a suitable dataset requires a lot of “annotated” unimodal data, as we need to ensure that each multimodal pair is related in a meaningful way. The situation is worse when we consider more complicated multimodal settings such as language–vision, where one-to-one or one-to-many correspondence between instances of the two datasets are required, due to the difficulty in categorising data such that commonality amongst samples is preserved within categories. See Figure 1 for an example from the CUB dataset (Welinder et al., a); although the same species of bird is featured in both image-caption pairs, their content differs considerably. It would be unreasonable to apply the caption from one to describe the bird depicted in the other, necessitating one-to-one correspondence between images and captions.
However, the scope of multimodal learning has been limited to leveraging the commonality between “related” pairs, while largely ignoring “unrelated” samples potentially available in any multimodal dataset—constructed through random pairing between modalities (Figure 3). We posit that if a distinction can be established between the “related” and “unrelated” observations within a multimodal dataset, we could greatly reduce the amount of related data required for effective learning. Figure 2 formalises this proposal. Multimodal generative models in previous work (Figure 2a) typically assumes one latent variable z that always generates related multimodal pair (x,y). In this work (Figure 2b), we introduce an additional Bernoulli random variable r that dictates the “relatedness” between x and y through z, where x and y are related when r = 1, and unrelated when r = 0.
∗work done while at Oxford
(a) Previous (b) Ours
Figure 2: Graphical models for multimodal generative process.
While r can encode different dependencies, here we make the simplifying assumption that the Pointwise Mutual Information (PMI) between x and y should be high when r = 1, and low when r = 0. Intuitively, this can be achieved by adopting a max-margin metric. We therefore propose to train the generative moels with a novel contrastive-style loss (Hadsell et al., 2006; Weinberger et al., 2005), and demonstrate the effectiveness of our proposed method from a few different perspectives: Improved multimodal learning: showing improved multimodal learning for various state-of-the-art multimodal
generative models on two challenging multimodal datasets. This is evaluated on four different metrics (Shi et al., 2019) summarised in § 4.2; Data efficiency: learning generative models under the contrastive framework requires only 20% of the data needed in baseline methods to achieve similar performance—holding true across different models, datasets and metrics; Label propagation: the contrastive loss encourages a larger discrepency between related and unrelated data, making it possible to directly identify related samples using the PMI between observations. We show that these data pairs can be used to further improve the learning of the generative model.

heading: 2 RELATED WORK
text: Contrastive loss Our work aims to encourage data-efficient multimodal generative-model learning using a popular representation learning metric—contrastive loss (Hadsell et al., 2006; Weinberger et al., 2005). There has been many successful applications of contrastive loss to a range of different tasks, such as contrastive predictive coding for time series data (van den Oord et al., 2018), image classification (Hénaff, 2020), noise contrastive estimation for vector embeddings of words (Mnih and Kavukcuoglu, 2013), as well as a range of frameworks such as DIM (Hjelm et al., 2019), MoCo (He et al., 2020), SimCLR (Chen et al., 2020) for more general visual-representation learning. The features learned by contrastive loss also perform well when applied to different downstream tasks — their ability to generalise is further analysed in (Wang and Isola, 2020) using quantifiable metrics for alignment and uniformity.
Contrastive methods have also been employed under a generative-model setting, but typically on generative adversarial networks (GANs) to either preserve or identify factors-of-variations in their inputs. For instance, SiGAN (Hsu et al., 2019) uses a contrastive loss to preserve identity for faceimage hallucination from low-resolution photos, while (Yildirim et al., 2018) uses a contrastive loss to disentangle the factors of variations in the latent code of GANs. We here employ a contrastive loss in a distinct setting of multimodal generative model learning, that, as we will show with our experiments and analyses promotes better, more robust representation learning. Multimodal VAEs We also demonstrate that our approach is applicable across different approaches to learning multimodal generative models. To do so, we first summarise past work on multimodal VAE into two categories based on the modelling choice of approximated posterior qΦ(z|x,y):
a) Explicit joint models: qΦ as single joint encoder qΦ(z|x,y). Example work in this area include JMVAE (Suzuki et al.), triple ELBO (Vedantam et al., 2018) and MFM (Tsai et al., 2019). Since the joint encoder require multimodal pair (x,y) as input, these approaches typically require additional modelling components and/or inference steps to deal with missing modality at test time; in fact, all three approaches propose to train unimodal VAEs on top of the joint model that handles data from each modality independently.
b) Factorised joint models: qΦ as factored encoders qΦ(z|x,y) = f ( qφx(z|x), qφy (z|y) ) .
This was first seen in Wu and Goodman (2018), as the MVAE model with f defined as a product of experts (POE), i.e. qΦ(z|x,y) = qφx(z|x)qφy (z|y)p(z), allowing for cross-modality generation without extra modelling components. Particularly, the MVAE caters to settings where data was not guaranteed to be always related, and where additional modalities were, in terms of information content, subsets of a primary data source—such as images and their class labels. Alternately, Shi et al. (2019) explored an approach that explicitly leveraged the availability of related/paired data, motivated by arguments from embodied cognition of the world. They propose the MMVAE model, which additionally differs from the MVAE model in its choice of posterior approximation—where f is modelled as the mixture of experts (MOE) of unimodal posteriors—to ameliorate shortcomings to do with precision miscalibration of the POE. Furthermore, Shi et al.
(2019) also posit four criteria that a multimodal VAE should satisfy, which we adopt in this work to evaluate the performance of our models.
Weakly-supervised learning Using generative models for label propagation (see § 4.5) is a form of weak supervision. Commonly seen approaches for weakly-supervised training with incomplete data include (1) graph-based methods (such as minimum cut) (Zhou et al., 2003; Zhu et al., 2003; Blum and Chawla, 2001), (2) low-density separation methods (Joachims, 1999; Burkhart and Shan, 2020) and (3) disagreement-based models (Blum and Mitchell, 1998; Zhou and Li, 2005; 2010). However, (1) and (2) suffers from scalability issues due to computational inefficiency and optimisation complexity, while (3) works well for many different tasks but requires training an ensemble of learners.
The use of generative models for weakly supervised learning has also been explored in Nigam et al. (2000); Miller and Uyar (1996), where labels are estimated using expectation-maximisation (EM) (Dempster et al., 1977) for instances that are unlabelled. Notably, models trained with our contrastive objective does not need EM to leverage unlabelled data (see § 4.5) to determine the relatedness of two examples we only need to compute a threshold (estimation of PMI) using the trained model.

heading: 3 METHODOLOGY
text: Given data over observations from two modalties (x,y), one can learn a multimodal VAE targetting pΘ(x,y, z) = p(z)pθx(x|z)pθy (y|z), where pθ(·|z) are deep neural networks (decoders) parametrised by Θ = {θx, θy}. To maximise the joint marginal likelihood log pΘ(x,y), one approximates the intractable model posterior pΘ(z|x,y) with a variational posterior qΦ(z|x,y), allowing us to optimise a variational evidence lower bound (ELBO), defined as
log pΘ(x,y) ≥ Ez∼ qΦ(z|x,y) [ log pΘ(z,x,y)
qΦ(z | x,y)
] = ELBO(x,y). (1)
This leaves open the question of how to model the approximatd posterior qΦ(z|x,y). As mentioned in § 2, there are two schools of thinking, namely explicit joint model such as JMVAE (Suzuki et al.) and factorised joint model including MVAE (Wu and Goodman, 2018) and MMVAE (Shi et al., 2019). In this work we demonstrate the effectiveness of our approach across all these models.

heading: 3.1 CONTRASTIVE LOSS FOR “RELATEDNESS” LEARNING
text: Where prior work always assumes (x,y) to be related, we introduce a relatedness variable r explicitly capturing this aspect. Our approach is motivated by the characteristics of pointwise mutual information (PMI) between related and unrelated observations across modalities:
Hypothesis 3.1. Let (x,y) ∼ pΘ(x,y) be a related data pair from two modalities, and let y′ denote a data point unrelated to x. Then, the pointwise mutual information I(x,y) > I(x,y′).
(a) Related (b) Unrelated
in PMI between related and unrelated pairs, i.e. I(x,y)− I(x,y′). We show in Appendix A that this is equivalent to maximising the difference between the joint marginal likelihoods of related and unrelated pairs (Figure 3a and 3b). This casts multimodal learning as max-margin optimisation, with the contrastive (triplet) loss as a natural choice of objective: LC(x,y,y′) = d(x,y)−d(x,y′) +m. Intuitively, LC attempts to make distance d between a positive pair (x,y) smaller than the distance between a negative pair (x,y′) by margin m. We adopt this loss to our objective by omitting margin m and replacing d with the (negative) joint marginal likelihood − log pΘ. Following Song et al. (2016), with N negative samples {y′i}Ni=1, we have
LC(x, Y ) = − log pΘ(x,y) + log N∑ i=1 pΘ(x,y ′ i). (2)
We choose to put the sum overN within the log following conventions in previous work on contrastive loss (van den Oord et al., 2018; Hjelm et al., 2019; Chen et al., 2020). As the loss is asymmetric, one can average over LC(y, X) and LC(x, Y ) to ensure negative samples in both modalities are accounted for—giving our contrastive objective:
LC(x,y)= 1
2 {LC(x, Y )+LC(y, X)}=−log pΘ(x,y)︸ ︷︷ ︸
1
+ 1
2 log N∑ x′i=1 pΘ(x ′,y) + log N∑ y′i=1 pΘ(x,y ′)  ︸ ︷︷ ︸
2
(3)
Note that since only the joint marginal likelihood terms are needed in (3), the contrastive learning framework can be directly applied to any multimodal generative model without needing extra components. We also show our contrastive framework applies when the number of modalities is more than two (cf. Appendix C).
Figure 4: log p(x,y) of imitation, unrelated digits and random noise, where p = N (mx,my).
Dissecting LC Although similar to the VAE (1), our objective (3) directly maximises pΘ(x,y) in 1 . LC by itself is not a effective objective as 2 in LC minimises pΘ(x,y), which can overpower the effect of 1 during training.
We intuit this phenomenon using a simple example in Figure 4 with natural images, showing the log likelihood of log p(x,y) in column 2, 3, 4 (green) on a Gaussian distribution N ((mx,my); c), with the
images in the first column of Figure 4 as means and with constant variance c. While achieving high log p(x,y) requires matching (mx,my) (col 2), we see both unrelated digits (col 3) and noise (col 4) can lead to (comparatively) poor joint log likelihoods. This indicates that the generative model need not generate valid, unrelated images to minimise 2 —generating noise would have roughly the same effect on log likelihood. As a result, the model can trivially minimise LC by generating noise that minimises 2 instead of accurate reconstruction that maximises 1 .
This learning dynamic is verified empirically, as we show in Figure 9 in Appendix D: optimising LC by itself results in the loss approaching 0 within the first 100 iterations, and both 1 and 2 takes on extremely low values, resulting in a model that generates random noise. Final objective To mitigate this issue, we need to ensure that minimising 2 does not overpower maximising 1 . We hence introduce a hyperparameter γ on 1 to upweight the maximisation of the marginal likelihood, with the final objective to minimise
LC(x,y) = −γ log pΘ(x,y)︸ ︷︷ ︸ 1
+ 1
2 log N∑ x′i=1 pΘ(x ′,y) + log N∑ y′i=1 pΘ(x,y ′)  ︸ ︷︷ ︸
2
, γ > 1. (4)
We conduct ablation studies on the effect of γ in Appendix E, noting that larger γ encourages better quality of generation and more stable training in some cases, while models trained with smaller γ are better at predicting “relatedness” between multimodal samples. We also note that optimising (4) maximises the pointwise mutual information I(x,y); see Appendix B for a proof.

heading: 3.2 OPTIMISING THE OBJECTIVE
text: Since in VAEs we do not have access to the exact marginal likelihood, we have to optimise an approximated version of the true contrastive objective in (4). In (5) we list a few possible candidates of estimators and their relationships to the (log) joint marginal likelihood:
ELBO ≤ E{zk}K1 ∼qΦ
[ log 1
K K∑ k=1 pΘ(zk,x,y) qΦ(zk | x,y) ] ︸ ︷︷ ︸
IWAE
≤ log pΘ(x,y) ≤ E{zk}K1 ∼qΦ log 2 √√√√ 1 K K∑ k=1 ( pΘ(zk,x,y) qΦ(zk | x,y) )2 ︸ ︷︷ ︸
CUBO
(5)
with the ELBO from Eq. (1), the importance weighted autoencoder (IWAE), a K-sample lower bound estimator that can compute an arbitrarily tight bound with increasing K (Burda et al., 2016), and the χ upper bound (CUBO), an upper-bound estimator (Dieng et al., 2017).
We now discuss our choice of approximation of log pΘ(x,y) for each term in equation (4):
1 Minimising this term maximises the joint marginal likelihood, which can hence be approximated with a valid lower-bound estimator (5); the IWAE estimator being the preferred choice.
2 For this term, we consider two choices of estimators:
1) CUBO: Since 2 of (4) minimises the joint marginal likelihoods, to ensure an upper-bound estimator to LC in (4), we need to employ an upper-bound estimator. Here we propose to use CUBO in (5). While such a bounded approximation is indeed desirable, existing upperbound estimators tend to have rather large bias/variance and can therefore yield poor quality approximations.
2) IWAE: We therefore also propose to estimate 2 with the IWAE estimator, as it provides an arbitrarily tight, low variance lower-bound (Burda et al., 2016) to log pΘ(x,y). Although this no longer ensure a valid bound on the objective, we hope that having a more accurate approximation to the marginal likelihood (and by extension, the contrastive loss) can affect the performance of the model positively.
We report results using both IWAE and CUBO estimators in § 4, denoted cI and cC respectively.

heading: 4 EXPERIMENTS
text: As stated in § 1, we analyse the suitability of contrastive learning for multimodal generative models from three persepctives—improved multimodal learning (§ 4.3), data efficiency (§ 4.4) and label propagation (§ 4.5). Throughout the experiments, we takeN = 5 negative samples for the contrastive objective, set γ = 2 based on analyses of ablations in appendix E, and take K = 30 samples for our IWAE estimators. We now introduce the datasets and metrics used for our experiments.

heading: 4.1 DATASETS
text: MNIST-SVHN The dataset is designed to separate conceptual complexity, i.e. digit, from perceptual complexity, i.e. color, style, size. Each data pair contains 2 samples of the same digit, one from each dataset (see examples in Figure 3a). We construct the dataset such that each instance from one dataset is paired with 30 instances of the same digit from the other dataset. Although both datasets are simple and well-studied, the many-to-many pairing between samples creates matching of different writing styles vs. backgrounds and colors, making it a challenging multimodal dataset. CUB Image-Captions We also consider a more challenging language-vision multimodal dataset, Caltech-UCSD Birds (CUB) (Welinder et al., b; Reed et al., 2016). The dataset contains 11,788 photos of birds, paired with 10 captions describing the bird’s physical characteristics, collected through Amazon Mechanical Turk (AMT). See CUB image-caption pair in Figure 1.

heading: 4.2 METRICS
text: Shi et al. (2019) proposed four criteria for multimodal generative models (Figure 5, left), that we summarise and unify as metrics to evaluate these criteria for different generative models (Figure 5, right). We now introduce each criterion and its corresponding metric in detail. (a) Latent accuracy (Figure 5a) Criterion: latent space factors into “shared” and “private” subspaces across modalities. We fit a linear classifier on the samples from z ∼ qΦ(z|x,y) to classify the information shared between the two modalities. For MNIST-SVHN, this can be the digit label as shown in Figure 5a (right). We check if l̂z is the same as the digit label of the original inputs x and y, with the intuition that extracting the commonality between x and y from latent representation using a linear transform, supports the claim that the latent space has factored as desired.
(b) Joint coherence (Figure 5b) Criterion: model generates paired samples that preserves the commonality observed in data. Again taking MNIST-SVHN as an example, this can be verified by taking pre-trained MNIST and SVHN digit classifiers and applying them on the multimodal observations generated from the same prior sample z. Coherence is computed by how often generations x̂ and ŷ classify to the same digit, i.e. whether l̂x̂ = l̂ŷ . (c) Cross coherence (Figure 5c) Criterion: model generates data in one modality conditioned on the other while preserving shared commonality. To compute cross coherence, we generate observations x̂ using latent from unimodal marginal posterior z ∼ qφ(z|y) and ŷ from z ∼ qφ(z|x). Similar to joint coherence, the criterion here is evaluated by predicting the label of the cross-generated samples x̂, ŷ using off-the-shelf MNIST and SVHN classifiers. In Figure 5c (right), the cross coherence is the frequency of which l̂x̂ = ly and l̂ŷ = lx. (d) Synergy coherence (Figure 5d) Criterion: models learnt across multiple modalities should be no worse than those learnt from just one. For consistency, we evaluate this criterion from a coherence perspective. Given generations x̂ and ŷ from z ∼ qΦ(z|x,y), we again examine if generated and original labels match; i.e. if l̂x̂ = ly = l̂ŷ = lx. See Appendix F for details on architecture. All quantitative results are reported over 5 runs. In addition to these quantitative metrics, we also showcase the qualitative results on both datasets in Appendix G and Appendix H.

heading: 4.3 IMPROVED MULTIMODAL LEARNING
text: Finding: Contrastive learning improves multimodal learning across all models and datasets. MNIST-SVHN See Table 1 (top, 100% data used) for results on the full MNIST-SVHN dataset. Note that for MMVAE, since the joint posterior qΦ is factorised as the mixture of unimodal posteriors qφx and qφy , the model never directly takes sample from the explicit form of the joint posterior. Instead, it takes equal number of samples from each unimodal posteriors, reflective of the equal weighting of the mixture. As a result, it is not meaningful to compute synergy coherence for MMVAE as it is exactly the same as the coherence of any single-way generation.
From Table 1 (top, 100% data used), we see that models trained on our contrastive objective (cI<MODEL> and cC-<MODEL>) improves multimodal learning performance significantly for all three generative models evaluated on the metrics. The results showcase the robustness of our approach from the perspectives of modelling choice and metric of interests. In particular, note that for the best performing model MMVAE, using IWAE estimator for 2 of Eq (4) (cI-MMVAE) yields slightly better results than CUBO (cC-MMVAE), while for the two other models the performance for different estimators are similar.
We also include qualitative results for MNIST-SVHN, including generative results, marginal likelihood table and diversity analysis in Appendix G.
CUB Following Shi et al. (2019), for the images in CUB, we observe and generate in feature space instead of pixel space by preprocessing the images using a pre-trained ResNet-101 (He et al., 2016). A nearest-neighbour lookup among all the features in the test set is used to project the feature generations of the model back to image space. This helps circumvent CUB image complexities to some extent—as the primary goal here is to learn good models and representations of multimodal data, rather than a focus on pixel-level image quality of generations.
The metrics listed in § 4.2 can also be applied to CUB with some modifications. Since bird-species classes are disjoint for the train and test sets, and as we show in Figure 1 contains substantial in-class variance, it is not constructive to evaluate these metrics using bird categories as labels. In Shi et al. (2019), the authors propose to use Canonical Correlation Analysis (CCA)—used by Massiceti et al. as a reliable vision-language correlation baseline—to compute coherence scores between generated image-caption pairs; which we employ (i.e. (b), (c), (d) in Figure 5) for CUB.
We show the results in Table 2 (top, 100% data used). We see that our contrastive approach (both cI-<MODEL> and cC-<MODEL>) is even more effective on this challenging vision-language dataset, with significant improvements to the correlation of generated image-caption pairs. It is also on this more complicated dataset where the advantages of using the stable, low-variance IWAE estimator are highlighted — for both MVAE and JMVAE, the contrastive objective with CUBO estimator suffers from numerical stability issues, yielding close-to-zero correlations for all metrics in Table 2. Results for these models are therefore omitted.
We also show the qualitative results for CUB in Appendix H.

heading: 4.4 DATA EFFICIENCY
text: Finding: Contrastive learning on 20% of data matches baseline models on full data.
We plot the quantitative performance of MMVAE with and without contrastive learning against the percentage of the original dataset used, as seen in Figure 6. We observe that the performance of contrastive MMVAE with the IWAE (cI-MMVAE, red) and CUBO estimators (cC-MMVAE, yellow) are consistently better than the baselines (MMVAE, blue), and that baseline performance using all related data is matched by the contrastive MMVAE using just 10—20% of data. The partial datasets used here are constructed by first taking n% of each unimodal dataset, then pairing to create multimodal datasets (§ 4.1)—ensuring it contains the requisite amount of “related” samples. In addition, we reproduce results generated from using 100% of the data in MNIST-SVHN and CUB (Tables 1 and 2) using only 20% of the original multimodal datasets (Tables 1 and 2 (bottom)). Comparing results between top vs. bottom in Tables 1 and 2 shows that this finding holds across the models, on both MNIST-SVHN and CUB datasets. This shows that the efficiency gains from a contrastive approach is invariant to VAE type, data, and metrics used, underscoring its effectiveness.

heading: 4.5 LABEL PROPAGATION
text: Finding: Generative models learned contrastively are good predictors of “relatedness”, enabling label propagation and matching baseline performance on full datasets, using only 10% of data.
Here, we show that our contrastive framework encourages a larger discrepancy between the PMI of related vs. unrelated data, as set out in hypothesis 3.1, allowing one to first train the model on a small subset of related data, and subsequently construct a classifier using PMI that identifies related samples in the remaining data. We now introduce our pipeline for label propagation in details.
Pipeline As showing in Figure 7, we first construct a full dataset by randomly matching instances in MNIST and SVHN, and denote the related pairs by Fr (full, related). We further assume access to only n% of Fr, denoted as Sr (small, related), and denote the rest as Fm, containing a mix of related and unrelated pairs. Next, we train a generative model g on Sr. To find a relatedness threshold, we construct a small, mixed dataset Sm by randomly matching samples across modalities in Sr. Given relatedness ground-truth for Sm, we can compute the PMI I(x,y) = log pΘ(x,y)− log pθx(x)pθy (y) for all pairs (x,y) in Sm and estimate an optimal threshold. This threshold can now be applied to the full, mixed dataset Fm to identify related pairs giving us a new related dataset Fr̂, which can be used to further improve the performance of the generative model g. Results In Figure 8 (a-e), we plot the performance of baseline MMVAE (blue) and contrastive MMVAE with IWAE (cI-MMVAE, red) and CUBO (cC-MMVAE, yellow) estimators for term 2 of Eq (4), trained with (solid curves) and without (dotted curves) label propagation. Here, the x-axis is the proportion in size of Sr to Fr, i.e. the percentage of related data used to pretrain the generative model before label propagation. We compare these results to MMVAE trained on all related data Fr (cyan horizontal line) as a “best case scenario” of these training regimes.
Clearly, label propagation using a contrastive model with IWAE estimator is helpful, and in general the improvement is greater when less data is available; Figure 8 also shows that when Sr is 10% of Fr, cI-MMVAE is competitive with the performance of baseline MMVAE trained on Fr.
For baselines MMVAE and cC-MMVAE however, label propagation hurts performance no matter the size of Sr, as shown by the blue and yellow curves in Figure 8 (a-e). The advantages of cIMMVAE is further demonstrated in Figure 8f, where we compute the precision, recall, and F1 score of relatedness prediction on Fm, for models trained on 10% of all related data. We also compare to a simple label-propagation baseline, where the relatedness of Fm is predicted using a siamese network (Hadsell et al., 2006) trained on the same 10% dataset. Notably, while the Siamese baseline in Figure 8f is a competitive predictor of relatedness, cI-MMVAE has the highest F1 score amongst all four, making it the most reliable indicator of relatedness. Beyond that, note that with the contrastive MMVAE, relatedness can be predicted without additional training and only requires a simple threshold computation directly computed using the generative model. The fact that the cI-MMVAE’s relatedness-prediction performance is the only one that matches the Siamese baseline strongly supports the view that the contrastive loss encourages generative models to utilise and better learn the relatedness between multimodal pairs; in addition, the poor performance of cC-MMVAE shows that the advantages of having a bounded estimation to the contrastive objective by using an
upper-bound for 2 is overshadowed by the high bias of CUBO, and that one may benefit more from choosing a low variance lower-bound like IWAE.

heading: 5 CONCLUSION
text: We introduced a contrastive-style objective for multimodal VAE, aiming at reducing the amount of multimodal data needed by exploiting the distinction between "related" and "unrelated" multimodal pairs. We showed that this objective improves multimodal training, drastically reduce the amount of multimodal data needed, and establishes a strong sense of "relatedness" for the generative model. These findings hold true across a multitude of datasets, models and metrics. The positive results of our method indicates that it is beneficial to utilise the relatedness information when training on multimodal data, which has been largely ignored in previous work. While we propose to utilise it implicitly through contrastive loss, one may consider relatedness as a random variable in the graphical model and see if explicit dependency on relatedness can be useful. It is also possible to extend this idea to Generative adversarial networks (GANs), by employing an additional discriminator that evaluates relatedness between generations across modalities. We will leave these interesting directions to be explored by future work.

heading: ACKNOWLEDGEMENTS
text: YS and PHST were supported by the Royal Academy of Engineering under the Research Chair and Senior Research Fellowships scheme, EPSRC/MURI grant EP/N019474/1 and FiveAI. YS was additionally supported by Remarkdip through their PhD Scholarship Programme. BP is supported by the Alan Turing Institute under the EPSRC grant EP/N510129/1. Special thanks to Elise van der Pol for helpful discussions on contrastive learning.

heading: A FROM POINTWISE MUTUAL INFORMATION TO JOINT MARGINAL LIKELIHOOD
text: In this section, we show that for the purpose of utilising the relatedness between mutlimodal pairs, maximising the difference between pointwise mutual information between related points x,y and unrelated points x,y′ is equivalent to maximising the difference between their log joint marginal likelihoods.
To see this, we can expand I(x,y)− I(x,y′) as
I(x,y)− I(x,y′) = [log pΘ(x,y)− log pΘ(x)− log pΘ(y)]− [log pΘ(x,y′)− log pΘ(x)− log pΘ(y′)] = log pΘ(x,y)− log pΘ(x,y′)︸ ︷︷ ︸
1
+ log pΘ(y ′)− log pΘ(y)︸ ︷︷ ︸
2
. (6)
In (6) the PMI difference is decomposed as two terms: 1 the difference between joint marginal likelihoods and 2 the difference between marginal likelihoods of different instances of y-s. It is clear that since 2 involves only one modality and only accounts for the difference in values between y and y′, it is not relevant to the relatedness of x, y.
Therefore, for the purpose of utilising relatedness information, we only need to maximise I(x,y)− I(x,y′) through maximising term 1 , i.e. the difference between joint marginal likelihoods.

heading: B CONNECTION OF FINAL OBJECTIVE TO POINTWISE MUTUAL INFORMATION
text: Here, we show that minimising the objective in (4) maximises the PMI between x and y:
L(x,y) = −γ log pΘ(x,y) + 1
2 log ∑ x′∈X pΘ(x ′,y) + log ∑ y′∈Y pΘ(x,y ′)  = −(γ − 1
2 ) log pΘ(x,y)−
1 2 log pΘ(x,y)∑ y′∈Y pΘ(x,y ′) ∑ x′∈X pΘ(y,x ′)
≈ −(γ − 1 2 ) log pΘ(x,y)− 1 2 log pΘ(x,y)
pΘ(x)pΘ(y)︸ ︷︷ ︸ I(x,y)
(7)
We see in (7) that minimising L can be decomposed to maximising both the joint marginal likelihood pΘ(x,y) and an approximation of PMI I(x,y). Note that since γ > 1, we can be sure that the joint marginal likelihood weighting γ − 12 is non-negative.

heading: C GENERALISATION TO M > 2 MODALITIES
text: In this section we show how the contrastive loss generalise to cases where number of modalities considered M is greater than 2.
Given observations from M modalities D = {X1, X2, · · · , Xm, · · · , XM}, where Xm denotes unimodal dataset of modalit m of size Nm, i.e. Xm = {x(i)m }Nmi=1. Similar to (2), we can write the assymetrical contrastive loss for any observation x(i)m from modality m, where negative samples are taken for all (M − 1) other modalities:
LC(x(i)m , Dm̃) = − log pΘ(x (i) 1:M ) + log M∑ d=1 d 6=m N∑ j=1 j 6=i pΘ(x (i) 1:(d−1),(d+1):M ,x (j) d ). (8)
We can therefore rewrite (3) as:
LC(x(i)1:M ) = 1
M M∑ m=1 LC(x(i)m , Dm̃) (9)
= − log pΘ(x(i)1:M ) + 1
M M∑ m=1 log M∑ d=1 d 6=m N∑ j=1 j 6=i pΘ(x (i) 1:(d−1),(d+1):M ,x (j) d )  , (10) where N is the number of negative samples, all log pΘ(x1:M ) are approximated by the following joint ELBO for M modalities:
log pΘ(x1:M ) ≥ Ez∼ qΦ(z|x1:M ) [ log pΘ(z,x1:M )
qΦ(z | x1:M )
] = ELBO(x1:M ). (11)
While the above gives us the true generalisation of (3), we note that the number of times where ELBO needs to be evaluated in (10) is O(M2N), making it difficult to implement this objective in practice, especially on more complicated datasets. We therefore propose a simplified version of the objective, where we estimate the second term of (10) with N sets of random samples from all modalities. Specifically, we can precompute the following M ×N random index matrix J :
J =  j11 j12 · · · j1N j21 j22 · · · j2N
... ...
jM1 jM2 · · · jMN  , (12) where each entry of J is a random integer taken from range [1, Nm]. We can then replace the second term of (10) random samples selected by the indices in J , giving us
LC(x(i)1:M ) ≈ − log pΘ(x (i) 1:M ) +
( log
N∑ n=1 pΘ(x (J1n) 1 ,x (J2n) 2 , · · ·x (JMn) M )
) . (13)
The number of times ELBO needs to be computed is now O(N), and is no longer relevant to the number of modalities M .
We can now also generalise the final objective in (4) to M modalities:
L(x(i)1:M ) = −γELBO(x (i) 1:M ) +
( logsumexp
n∈N ELBO(x(J1n)1 ,x (J2n) 2 , · · ·x (JMn) M )
) . (14)

heading: D THE INEFFECTIVENESS OF TRAINING WITH LC ONLY
text: We demonstrate why training with the contrastive loss proposed in (3) is ineffective, and why additional ELBO term is needed for the final objective. As we show in Figure 9, when training with LC only, while the contrastive loss (green) quickly drops to zero, both term 1 and 2 in (3) also reduces drastically. This means the joint marginal likelihood of any generation log pΘ(x,y) is small regardless the relatedness of (x,y).
In comparison, we also plot the training curve for model trained on the final objective in (4), which upweights term 1 in (3) by γ. We see in Figure 10 that by setting γ = 2, the joint marginal likelihood (yellow and blue curve) improves during training, while LC (green curve) gets minimised.
E ABLATION STUDY OF γ
In § 3, we specified that γ needs to be greater than 1 to offset the negative effect of minimising ELBO through term 2 in (4). Here, we study the effect of γ in details.
Figure 11 compares latent accuracy, cross coherence and joint coherence of MMVAE on MNISTSVHN dataset trained on different values of γ. Note that here we only consider cases where γ ≥ 1, since the minimum value of γ is 1. In this case, the loss reduces to the original contrastive objective in (3).
A few interesting observations from the plot are as follows: First, when γ = 1, the model is trained using the contrastive loss only, and as we showed is an ineffective objective for generative model learning. This is verified again in Figure 11 — when γ = 1, both coherences and latent accuracies take on extremely low values; interestingly, there is a significant boost of performance across all metrics by simply increasing the value of γ from 1 to 1.1; after that, as the value of γ increases, performance on most metrics decreases monotonically (joint coherence being the only exception), and eventually converges to baseline MMVAE (dotted lines in Figure 11). This is unsurprising, since the final objective in (4) reduces to the original joint ELBO as γ approaches infinity.
Figure 11 seems to suggest that 1.1 is the optimal value for hyperparameter γ, however close inspection of the qualitative generative results shows that this might not be the case. See Figure 12 for a comparison of the model’s generation between MMVAE models trained on (from left to right) γ = 1.1, γ = 2 and γ = +∞ (i.e. original MMVAE). Although γ = 1.1 yields model with high coherence scores, we can clearly see from the left-most column of Figure 12 that the generation of the model seems deprecated, especially for the SVHN modality, where the backgrounds of model’s generation appear to be unnaturally spotty and deformed. This problem is mitigated by increasing γ — as shown in Figure 11, the image generation quality of γ = 2 (middle column) is not visibly different from that of γ = +∞ (right column).
To verify this observation, we also compute the marginal log likelihood log pΘ(x,y) to quantify the quality of generations. We compute this for all γs considered in Figure 11, and take the average over the entire test set. From the results in Figure 13, we can see a significant increase of the log likelihood between γ = 1.1 to γ = 1. This gain in image generation quality then slows down as γ further increases, and as all other metrics converges to the original MMVAE model.

heading: F ARCHITECTURE
text: We use architectures listed in Table 3 for the unimodal encoder and decoder for MMVAE, MVAE and JMVAE. For JMVAE we use an extra joint encoder, the architecture of which is described in Table 4.

heading: G QUALITATIVE RESULTS ON MNIST-SVHN
text: G.1 GENERATIVE RESULTS & MARGINAL LIKELIHOODS ON MNIST-SVHN G.1.1 MMVAE
G.1.2 MVAE MVAE cI-MVAE cC-MVAE
G.1.3 JMVAE JMVAE cI-JMVAE cC-JMVAE
G.2 GENERATION DIVERSITY To further demonstrate that the improvements from our contrastive objective did not come at a price of sacrifising generation diversity, in Figure 17 we show histograms of the number of examples generated for each class with samples from prior. Similar to how we compute joint coherence, the class label of generated images are determined using classifiers trained on the original MNIST and SVHN dataset.
We can see that the contrastive models (cI-∗ and cI-∗) are capable of generating examples from different classes, and for MMVAE and MVAE the histogram of contrastive models are more uniform than the original models (less variance between class counts).

heading: H QUALITATIVE RESULTS ON CUB
text: The generative results of MMVAE, cI-MMVAE and cC-MMVAE on CUB Image-Caption dataset are as shown in Figure 18, Figure 19 and Figure 20. Note that for the generation in the vision modality, we reconstruct and generate features from ResNet101 and perform nearest neighbour search in all the features in train set to showcase our generation results.

Review summaries:
The paper tries to minimize the difference of the PMI between related and unrelated pairs of multimodal data but arrives at a very different objective with many approximations. It can be plugged into existing VAE based methods and improve learning performance and data efficiency. My major concern is about the derivation and the connection between motivation and the final objective.

 The paper proposes a contrastive objective that (1) minimizes the distance between "related" samples while (2) maximizing the distance between randomly paired samples. Existing multimodal VAEs optimize (1) via different multimodal ELBOs. The novelty lies in the optimization of (2) which can further benefit from unimodal samples for which no "related" samples of the other modality are available---this can be viewed as a semi-supervised approach for weakly-supervised multimodal data.  For the estimation of (2), the paper experiments with two different estimators, IWAE and CUBO.

This work presents a generative model for multimodal learning. The paper maximizes or minimizes the pointwise mutual information between data from two modalities considering a novel random variable relatedness to dictates if data are related or not.  This is realized by casting multimodal learning as max-margin optimization with the contrastive loss for the objective. For the optimization, the paper considers the IWAE estimator. As per the experiments, the paper considers MNIST-SVHN and CUB Image-Captions dataset and perform evaluations across four metrics. Using the experiments, the paper demonstrates that the proposed approach improves multimodal learning, data-efficient learning, and label propagation.