Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Models
Authors: Yuge Shi
Brooks Paige
Philip Torr
Siddharth N
Venue: ICLR 2021 Poster (2021)
Keywords: Deep generative model
multi-modal learning
representation learning

Abstract: Multimodal learning for generative models often refers to the learning of abstract concepts from the commonality of information in multiple modalities, such as vision and language. While it has proven effective for learning generalisable representations, the training of such models often requires a large amount of related multimodal data that shares commonality, which can be expensive to come by. To mitigate this, we develop a novel contrastive framework for generative model learning, allowing us to train the model not just by the commonality between modalities, but by the distinction between "related" and "unrelated" multimodal data. We show in experiments that our method enables data-efficient multimodal learning on challenging datasets for various multimodal VAE models. We also show that under our proposed framework, the generative model can accurately identify related samples from unrelated ones, making it possible to make use of the plentiful unlabeled, unpaired multimodal data.

Published as a conference paper at ICLR 2021
RELATING BY CONTRASTING: A DATA-EFFICIENT FRAMEWORK FOR MULTIMODAL DGMs
Yuge Shi', Brooks Paige?,, Philip H.S. Torr' & N. Siddharth 1,3,4
1 University of Oxford
University College London University of Edinburgh 'The Alan Turing Institute
yshi frobbts ax.ac. uk
ABSTRACT
Multimodal learning for generative models often refers to the learning of abstract concepts from the commonality of information in multiple modalities, such as vision and language. While it has proven effective for learning generalisable representations, the training of such models often requires a large amount of "related" multimodal data that shares commonality, which can be expensive come by. To mitigate this, we develop a novel contrastive framework for generative model learning, allowing to train the model not just by the commonality between modalities, but by the distinction between "related" and "unrelated" multimodal data. We show in experiments that our method enables data-efficient multimodal learning on challenging datasets for various multimodal variational autoencoder (VAE) models. We also show that under our proposed framework, the generative model can accurately identify related samples from unrelated ones, making it possible to make use of the plentiful unlabeled, unpaired multimodal data.
INTRODUCTION
To comprehensively describe concepts in the real world, humans collect multiple perceptual signals of the same object such as image, sound, text
Category: Laysan
"A sized
and video. We refer to each of these media as a modality, and collection of different media featuring the same underlying concept characterised multimodal data. Learning from multiple modalities has been shown to yield more generalisable representations (Zhang et al.) (2020) Guo et al., 2019; Yildirim 2014), different modalities often complimentary in content while overlapping for common abstract concept.
with glosing
white body and large orange
"A clgar- shaped white with long brown wings hooked Long bill."
Figure 1: Multimodal data Despite the motivation, is worth noting that the multimodal framework from the CUB dataset is not exactly data-efficient- ttteefffiiettccnstuuttnnnn a suitable dataset requires a lot of "annotated" unimodal data, as we need to ensure that each multimodal pair is related in meaningful way. The situation is worse when we consider more complicated multimodal settings such as language-vision, where one-to-one or me-to-many correspondence between instances of the two datasets are required, due to the difficulty in categorising data such that commonality amongst samples is preserved within categories. See Figure 1 for an example from the CUB dataset (Welinder al. a): although the same species of bird is featured in both image-caption pairs, their content differs considerably. It would be unreasonable to apply the caption from one to describe the bird depicted the other, necessitating one-to-one correspondence between images and captions.
However, the scope of multimodal learning has been limited to leveraging the commonality between "related" pairs, while largely ignoring "unrelated" samples potentially available in any multimodal constreteo through random pairing between modalities (Figure 3). We posit that if a distinction can be established between the "related" and "unrelated" observations within multimodal dataset, we could greatly reduce the amount of related data required for effective learning. Figure 2 formalises this proposal. Multimodal generative models in previous work (Figure 2a typically assumes one latent variable z that always generates related multimodal pair (æ, y). In this work (Figure 2b), we introduce an additional Bernoulli random variable that dictates the "relatedness" between x and y through 2, where x and y are related when = 1, and unrelated when r = 0.
"wor done while at Oxford

---

Published as a conference paper at ICLR 2021
While can encode different dependencies, here make the sim- plifying assumption that the Pointwise Mutual Information (PMI) between and y should be high when r 1, and low when O. Intuitively, this can be achieved by adopting a max-margin metric. We therefore propose contrastive-style loss (Hadsell et al. 2006, Weinnerger et al. 2005), and demonstrate the effectiveness of our proposed method from few different perspectives: Improved multimodal learning: showing improved multimodal learning for various state-of-the- art multimodal generative models two challenging multimodal datasets. This is evaluated on four different metrics (Shi et al. 2019) summarised in 4.2 Data efficiency: learning generative models under the contrastive framework requires only 20% of the data needed in baseline methods to achieve similar cerformance-holdimn true across different models, datasets and metrics; Label propagation: the contrastive loss encourages larger discrepency between related and unrelated data, making it possi- ble to directly identify related samples using the PMI between observations. We show that these data pairs can be used to further improve the learning of the generative model.
génarative modal inference model
train the generative moels with novel
y
(a) Previous (b) Ours
Figure 2: Graphical models for multimodal generative process.
RELATED WORK
Contrastive loss Our work aims to encourage data-efficient multimodal generative modell learning
using a popular representation learning metric- contrastive loss (Hadsell et al. 2006 Weinberger
al. 2005). There has been many successful applications of contrastive loss to a range of different tasks, such contrastive predictive coding for time series data (van den Oord al. 2018), image classification (Hénaff) 2020), noise contrastive estimation for vector embeddings of words (Mnih and Kavukcuoglu 2013). as well as a range of frameworks such as DIM (Hjelm et al. 2201)). MoCo (He et al., (020)), SimCLR (Chen et al. 2020) for more general risual-representation learning. The features learned by contrastive loss also perform well when applied different downstream tasks their ability to generalise is further analysed in (Wang and Isola 2020) using quantifiable metrics for
alignment and uniformity.
Contrastive methods have also been employed under generative- setting, but typically on generative adversarial networks (GANs) to either preserve or identify factors-of-vvaiation in their inputs. For instance, SiGAN (Hsu et al., 2019) uses contrastive loss to preserve identity for face- image hallucination from low-resolution photos, while (Yildirim et al. [2018) uses a contrastive loss to disentangle the factors of variations in the latent code of GANs. We here employ contrastive loss in a distinct setting of multimodal generative model learning, that, as we will show with our experiments and analyses promotes better, more robust representation learning.
Multimodal VA We also demonstrate that our approach is applicable across different approaches to learning multimodal generative models. To do so, we first summarise past work on multimodal VAE into two categories based on the modelling choice of approximated posterior qu (z a. y):
a) Explicit joint models: Φ4 as single joint encoder qa (z le, y).
Example work in this area include JMVAE (Suzuki et al.), triple ELBO (Vedantam et al. 2018) and MFM (sai et al. 2019). Since the joint encoder require multimodal pair (z, y) as imput, these approaches typically require additional modelling components and/or infer- ence steps to deal with missing modality at test time; in fact, all three approaches propose to train unimodal VAEs on top of the joint model that handles data from each modality independently.
b) Factorised joint models: 44 as factored encoders q (z /e. y) f (gs, (z)ar), L6, (=ly)) This was first seen in Wu and Goodman (2018), as the MVAE model with f defined as a product of experts (PoE), i.c. 9o (z x. y) To (= x ) (z ([))pp(()) allowing for cross- crossmmodality generation without extra modelling components. Particularly, the MVAE caters to settings where data was not guaranteed to be always related, and where additional modalities were, in terms of information content, subsets of a primary data source-such as images and their class labels.
Alternately, Shi et al. (2019) explored an approach that explicitly leveraged the availability of related/paired data, motivated by arguments from embodied cognition of the world. They propose the MMVAE model, which additionally differs from the MVAE model in its choice of posterior approximation-wwe where ff is modelled as the mixture of experts (MoE) of unimodal posteriors- to ameliorate shortcomings to do with precision miscalibration of the POE. Furthermore, (Shi et al.
2

---

Published as a conference paper at ICLR 2021
2019) also posit four criteria that a multimodal VAE should satisfy, which we adopt in this work to evaluate the performance of our models.
learning Using generative models for label propagation (see { 4.5) is form of weak supervision. Commonly seen approaches for weakly-supervised training with incomplete data include (1) graph-base based methods (such as minimum cut) (Zhou et al. [2003] Zhuu et al. [2003] Blum and
Chawla 2001), (2) low-density separation methods (Joachims] [999] Burk and Shan 2020) and
(3) disagreementtbased models (Blum and Mitchell 1998, Zhou and Lij [205]] 2010). However, (1)
and (2) suffers from scalability issues due to computational inefficiency and optimisation complexity,
while (3) works well for many different tasks but requires training an ensemble of learners.
The use of generative models for weakly supervised learning has also been explored in Nigam et al. (2000); Miller and Uyar (1996). where labels are estimated using (Dempster et 1977)
(EM) > for instances that are unlabelled. Notably, models trained with our contrastive objective does not need EM to leverage unlabelled data (see S 4.5) to determine the relatedness of two examples we only need to compute a threshold (estimation of PMI) using the trained model.
METHODOLOOGY
Given data over observations from two modalties (x, y). one can learn a multimodal VAE tar- getting pe (æ,, y, z) - p(z)pp__ (as (z)=))o, (y]=), where p=(- are deep neural networks (decoders) parametrised by θ = {0.z J. To maximise the joint marginal likelihood log pe (æ, y), one approxi- mates the intractable model posterior pe æz y) with variational posterior qu- (z az y), allowing us to optimise a variational evidence lower bound (ELBO), defined as
pee (e, x. y)
log pe y) 2 (x log
ELBO(x; y).
(1)
g x.
This leaves open the question of how to model the approximatd posterior qu x. y). As mentioned in 2, there two schools of thinking. namely explicit joint and factorised joint model including MVAE (Wu and Goodman (018) and MMVAE (Shi et al. 2019). In this work we demonstrate the effectiveness of our approach across all these models.
joint model such JMVAE (uzuki al.)
3.1 CONTRASTIVE LOSS FOR "RELATEDNESS" LEARNING
Where prior work al always assumes (at, y to be related, we introduce a relatedness variable r ex- plicitly capturing this aspect. Our approach is motivated by the characteristics of pointwise mutual information (PMI) between related and unrelated observations across modalities:
Hypothesis 3.1. Let (x, y) pa( y) be a related data pair from two modalities, and let y denote a data point unrelated to x. Then, the pointwise mutual information I (xx y) > 1(x, y').
Note that the PMI measures the statistical dependence between values (1,, y). which for a joint distribution p(z, y) is defined as [(x: y) = log p(z)p()) Hypothesis 3.1 should be fairly uncontroversial assumption for true gen- erative models: we say simply that under the joint distribution for related data pe (ar. y). the PMI between related points (æ. y) is stronger than that between unrelated points (ær, y²). In fact, we demonstrate in S 4.5 that for well-trained generative model, PMI is good indicator of relatedness, enabling pairing of random multimodal data by thresholding the PMI esti- mated by the trained model. It is therefore possible to leverage relatedness while training parametrised generative model by maximising the difference in PMI between related and unrelated pairs, i.e. I(x: 3) [(x y¹). We show in Appendix A that this is equivalent to maximising the difference between the joint marginal likelihoods of related and unrelated pairs (Figure 3al and (3bp. This casts multimodal learning as max-margin optimisation, with the contrastive (triplet) loss as natural choice of objective: (x, y, y d(æ, y) d(æ, y') + m. Intuitively, Lo attempts make distance between positive pair (x, y) smaller than the distance between negative pair (æt, y by margin m. We adopt this loss to our objective by omitting margin m and replacing d with the (negative) joint marginal likelihood log pa. Following Song al. (2016), with N negative samples {1(). 141 we have
0
0
p(z.3)
4
4
5
5
6
(b) Unrelated Figure 3: Constructing re-
(n) Relaned
lated & unrelated samples
N
Lo(x, Y) log pe (xt y) + log
pe (x, v.).
3

---

Published as a conference paper at ICLR 2021
We choose to put the loss (van den Oord et al., 2018 Hjelm et al. 2019; Chen et al. 2020). As the loss is asymmetric, one average Le y X and Lo(x, ensure negative samples in both modalities are accounted for-giving our contrastive objective:
N within the log following conventions in previous work contrastive
N pa a v)
Lolæ, y) 2 +) -log==-ooppppeee
pa(
log
log pel (a: v')
(3)
Note that since only the joint marginal likelihood terms are needed in (3). the contrastive learning framework can be directly applied to any multimodal generative model without needing extra components. We also show our contrastive framework applies when the number of modalities is more than two (cf. [Appnndix C).
mz.
Suttation
Dissecting Le Although similar the VAE (1) objective directly maximises (a y) in D. Lo by itself is not a effective objective as 2 in Le minimises pe (æ, y), which can overpower the effect of during training.
5
5
5
5
48
logp(s. p) -4.74 -80.17, -99.75
We intuit this phenomenon using simple example Figure with natural images, showing the log likelihood of log p(æ, in column
Figure 4: log,(æ, y) of imitation, unrelated digits and random noise, where p =N(mz, my).
2, 3, (green) on a Gaussian distribution ((m,a my); c), with the images in the first column of Figure 4]as means and with constant variance While achieving high log p(x. (z.3) y) requires matching (m. myy) (col 2), we see both unrelated digits (col 3) and noise (col 4) can lead to (comparatively) poor joint log likelihoods. This indicates that the generative model need not generate valid, unrelated images to minimise 22 -enneratingg noise would have roughly same effect log likelihood. As result, the model can trivially minimise Lo by generating noise that minimises 2 instead of accurate reconstruction that maximises
This learning dynamic is verified empirically,
show in Figure 9 in Appendix D optimising Le by itself results in the loss approaching within the first 100 iterations, and both (I and 2 takes on extremely low values, resulting in model that generates random noise.
Final objective To mitigate this issue, we need to ensure that minimising 2 does not overpower maximising I We hence introduce hyperparameter y on I to upweight the maximisation of the marginal likelihood, with the final objective to minimise
Lc(x, v) -y dogre(as (2, log
pe (e 4) + log
pe (e v')
7>14 (4)
V
We conduct ablation studies on the effect of in Appendix E. noting that larger y encourages better quality of generation and more stable training in some cases, while models trained with smaller are better at predicting "relatedness" between multimodal samples. We also note that optimising (4) maximises the pointwise mutual information [(x, y); see Appendix B for a proof.
3.2 OPTIMISING THE ORJECTIVE
Since in VAES we do not have access to the exact marginal likelihood, we have to optimise an approximated version of the true contrastive objective in (4). In (5) we list few possible candidates of estimators and their relationships to the (log) joint marginal likelihood:
Pol Se. v)
(= (3iz
2
ELBO & E Ef age
log
< log (at, y) & -o
log
(5)
do
(2
IWAE
CU6O
with the ELBO from Eq. the importance weighted autoencoder (IWAE), a K- Kasmpll lower bound estimator that can compute arbitrarily tight bound with increasing K (Burda et al. 2016). and the upper bound (CUBO), an upper-bound estimator (Dieng et al. 2017).
We now discuss choice of approximation of log (ar y) for each term in equation
4

---

Published as a conference paper at ICLR 2021
Minimising this term maximises the joint marginal likelihood, which can hence be approximated
with a valid lower-boune estimator (5 ; the IWAE estimator being the preferred choice.
2 For this term, consider two choices of estimators:
1) CUBO: Since (2) of (4h minimises the joint marginal likelihoods, to ensure an upper-bound estimator to Le in 44, we need to employ upper-bound estimator. Here propose to use CUBO in (5). While such a bounded approximation is indeed desirable, existing upper- bound estimators tend have rather large bias/variance and can therefore yield poor quality approximations.
2) IWAE: We therefore also propose to estimate ② with the IWAE estimator, as provides an arbitrarily right, low variance lower-bound (Burda et al. 2016) to log per y). Although this no longer ensure valid bound on the objective, we hope that having a more accurate approximation to the marginal likelihood (and by extension, the contrastive loss) can affect the performance of the model positively.
We report results using both IWAE and CUBO estimators in
4 denoted eI and cC respectively.
EXPERIMENTS
As stated in S I. analyse the suitability of contrastive learning for multimodal generative models from three persepetives-imprroed multimodal learning $ 4.3 data efficiency (8 4.4) and label propagation (B 4.3 Throughout the experiments,
take N 5 negative samples for the contrastive objective, set y = 2 based on analyses of ablations in aappnndix E, and take K = 30 samples for our IWAE estimators. We now introduce the datasets and metries used for our experiments.
4.1 DATASETS
MNIST-SVHN The dataset is designed to separate conceptual complexity, i.e. digit, from percep- tual complexity, i.c. color, style, size. Each data pair contains 2 samples of the same digit, one from each dataset (see examples in Figure 3ap. We construct the dataset such that each instance from one dataset is paired with 30 instances of the same digit from the other dataset. Although both datasets are simple and well-studied, the many-to-many pairing between samples creates matching of different writing styles vs. backgrounds and colors, making it a challenging multimodal dataset.
CUB Image-Captions We also consider more challenging language- vision multimodal dataset, Caltech-UCSD Birds (CUB) (Welinder et al., b Reed et al. 2016). The dataset contains 11,788 photos of birds, paired with 10 captions describing the bird's 's physical characteristics, collected through Amazon Mechanical Turk (AMT). See CUB image-caption pair in Figure 1
singuage-vision
4.2 METRICS
Shi et al. (2019) proposed four criteria for multimodal generative models Figure 5 left), that we summarise and unify as metries to evaluate these criteria for different generative models (Figure 5 right). We now introduce each criterion and its corresponding metric in detail.
(a) Latent accuracy (Figure 5a Criterion: latent space factors into "shared" and "private" sub- spaces across modalities. We fit a linear classifier on the samples from z ~ qu qu x. y) to classify the information shared between the two modalities. For MNIST-SVHN, this can be the digit label as shown in Figure 5a (right). We check if i, is the same the digit label of the original inputs and y. with the intuition that extracting the commonality between æ and y from latent representation using a linear transform, supports the claim that the latent space has factored as desired.
by
by
Labert spac
Raralom varidihes
Criterion
Metric
Criterion
Metric s(a)
Metric lesi (ik)))))
Criterion
Linnar classiTier
(a) Latent Eactorisation
(b) Coherent joint generation
(c) Coherent cross generation
(d) Symergy
Figure 5: Left of each pair: Four criteria for multi-modal generative models; image adapted from Shi et al. 2019). Right of each pair: Four metrics to evaluate the model's performance on criterion in corresponding row.
5

---

Published as a conference paper at ICLR 2021
Criterion: model generates paired samples that preserves the com-- monality observed in data. Again taking MNIST-SVHN as an example, this can be verified by taking pre-trained MNIST and SVHN digit classifiers and applying them on the multimodal observations generated from the same prior sample 2. Coherence is computed by how often generations é and ý classify to the same digit, i.. whether 17 -
(b) Joint coherence (Figure 5b)
(c) Cross coherence (Figure 5c) Criterion: model generates data in one modality conditioned on the other while preserving shared commonality. To compute cross coherence, we generate observations t using latent from unimodal marginal posterior z ~ (z(y))) and ý from 2 ~ (0(2|1z), Similar to joint coherence, the criterion here is evaluated by predicting the label of the cross-generated samples i y using off-the-shelf MNIST and SVHN classifiers. In Figure 5c (right), the cross coherence is the frequency of which l = ly and ig = ln-
(d) Synergy coherence (Figure 5d) Criterion: models learnt across multiple modalities should be no worse than those learnt from just one. For consisteney, we evaluate this criterion from a coherence perspective. Given generations 2 and ý from x ~ qa (z x., y), we again examine if generated and original labels match; i.e. if ( 11yy-1y
lz.
See Appendix F] for details on architecture. All quantitative results are reported over runs. In addition to these quantitative metrics, we also showcase the qualitative results on both datasets in Appendix G and Appendix H
4.3 IMPROVED MULTIMODAL LEARNING
Finding: Contrastive learning improves multimodal learning across all models and datasets.
MNIST-SVHN See Table (top. 100% data used) for results the full MNIST-SVHY dataset. Note that for MMVAE, since the joint posterior qa is factorised as the mixture of unimodal posteriors Id., and g the model never directly takes sample from the explicit form of the joint posterior. Instead, it takes equal number of samples from each unimodal posteriors, reflective of the equal weighting of the mixture. As a result, it is not meaningful compute synergy coherence for MMVAE as it is exactly the same as the coherence of any single-way generation.
From Table (top, 100% data used), we see that models trained on our contrastive objective (cl- <MODEL> and eC-<MODEL>) improves multimodal learning performance significantly for all three generative models evaluated on the metrics. The results showcase the robustness of our approach from the perspectives of modelling choice and metric of interests. In particular, note that for the best performing model MMVAE, using IWAE estimator for 2 of Eq (4) (cI-MMVAE) yields slightly better results than CUBO (eC-MMVAE), while for the two other models the performance for different estimators are similar.
We also include qualitative results for MNIST-S VVHN, including generative results, marginal likeli- hood table and diversity analysis in [Appendix G
Table 1: Evaluation of baselines MMVAE, MVAE, JMVAE and their contrastive variations (cI-<MODEL>. cC-<MODEL> for the IWAE and CUBO estimators used in Eq respectively), MNIST())SSVH(()))) dataset, using 100% (top) and 20% (bottom) of data.
Latent accuracy (%)
Joint
Cross coherence (%)
Synergy coherence (%)
Data
Methods
Coherence (%6
S-M
M-S
joint-
joint->S
MMVAR
92.48 (10:1)
79.03 (L)17)
42.32 (1:17)
70.77 (1:31)
85 (2209)
d--MMAE
(1434)
(1032)
(1ose)
79.66 (1ss9)
92.67 (1029)
cC-MMAA
93.10 (s027)
80.88 (z0m0)
45.46 (=928)
79.34 (sass)
92.35 (#0.4)
100% of data used
MVAF
91.65 (10m)
64.12 (1410)
9.42 (1771)
(1ow)
21.88 (1321)
64.60 (1929)
52.91 (L4M))
MMAV
96.97 (10M)
75.94 (1620)
15.23 (2)0))
10.85 (1ze))
27.70 (1200)
85.07 (1225)
75.67 (1:41)
cC-MAAR
97.42 (10.0)
(1201)
8.85 (111)
(1771)
30.03 (1240
75.25 (1sm)
69.42 (L
IMVAR
84.45 (LOAT)
$7.98 (L))))
42.18 (L))8)
49.63 (1/
54.98 (1:10)
85.77 (Law)
68. (L)
eI-JMVE
(1:140
(11.0)
48.95 (1237)
(1:Aa)
(1:1d)
93.45 (:0.2)
(20w)
.C.MMVAR
83.67 (L.44)
66.64 (12ve)
47.27 (L417)
59.73 (L.1A)
69.49 (1211)
91.21 (1439)
84.07 (L
MMVAE
88.54 (+0337
68.90 (=1.29)
37,71 (#000)
59.52 (#025)
76.33 (#222)
cI-MMVAE
91.64 (1:0)
73,02 (=000)
42.74 (1934)
69.51 (1AM8)
86.75 (2026)
cC-MMVAE
92.10 (Lo.9)
71.29 (1))
40.77 (LoWs)
68.43 (Lo90)
86.24 (16m)
20% of data used
MVAE
90,29 (+0.57)
33.44 (en26)
10.88 (#9.5)
8.72 (4092)
12.12 (#120)
42.10 (=:22)
44,95 (=:52)
cL-MAAE
93.72 (17.8)
56.74 (1777)
12.79 (26xr)
14.18 (1719)
20.23 (1+15)
75.36 (=:00)
(L-49)
cC-MVAE
92.74 (#297)
52.99 (6AI5)
17.95 (e12.52)
14.70 (+:65)
24.90 (#577)
56.86 (#1e5))
54,28 (293s)
JMVAB
77.53 (2023)
52.55 (#2:8)
26.37 (ens))
42.58 (4532)
41.44 (#220)
85.07 (2970)
51.95 (=228)
CI-MVVA
77.57 (2442)
57.91 (#118)
32.58 (#:39)
51.85 (#1.27
47.92 (#nut)
92.54 (=2.1)
67.01 (#472)
CCJMMVAB
81.11 (+2.%)
57.85 (#227)
34.00 (#7.19
50.73 (4045)
56.89 (3x.1)
88.36 (+4.36)
68.49 (=4x2)

---

Published as a conference paper at ICLR 2021
Following Shi et al. (2019), for the images in CUB, observe and generate in feature space instead of pixel space by preprocessing the images using a pre- trtinadded ResNet-101 (He et al.) 2016). A nearest-neighhour lookup among all the features in the test set is used to project the feature generations of the model back to image space. This helps circumvent CUB image complexities to some extent-as as the primary goal here is to learn good models and representations of multimodal data, rather than a focus on pixel-level image quality of generations.
CUB
The metries listed $ 4.2 can also be applied to CUB with some modifications. Since bird- specieseeeessss classes are disjoint for the train and test sets, and as we show in Figure conttiiss substantial in-class variance, it is not constructive to evaluate these metrics using bird categories as labels. In Shi et al. (2019). the authors propose to use Canonical Correlation Analysis (CCA)-used by Massiceti et al. as a reliable iisoonlaagguage correlation baseline- to compute coherence scores between generated image-caption pairs; which we employ (i.e. (b), (c), (d) in Figure 5) for CUB.
We show the results in Table 2 (top, 100% data used). We see that our contrastive approach (both cI-<MODEL> and cC-<MODEL>) is even more effective on this challenging vision-language dataset, with significant improvements to the correlation of generated image-caption pairs. It is also on this more complicated dataset where the advantages of using the stable, low-variance IWAE estimator are highlighted - for both MVAE and JMVAE, the contrastive objective with CUBO estimator suffers from numerical stability issues, yielding close-to-zero correlations for all metrics in Table Results for these models are therefore omitted.
We also show the qualitative results for CUB in Appendix H.
Table 2: Evaluation of baseline MMVAE, MVAE, JMVAE and their contrastive variations (cI-<MODEL> and dataset, using 100% (top) and 20% (bottom) of data.
cC-<MODEL>> on CUB (agg((im))ccationn
Joint Coherence (CCA)
Cross coherence (CCA)
Synergy coherence (CCA)
Data
Methods
img -cap
cap-+ ing
joint ing
joint capp
MMVAE
0.212 (#2946-2)
0.154 (#708e4)
0,244 (#S.te-3)
el MMVAE
0.314 (1472-1)
0,188 (1440-4)
0.334 (111382)
100% of data used
cC-MMVAE
0,263 (ar47-a)
0,244 (=224-3)
0.369 (=7Me-))
MVAE
0.075
(187162)
-0.008 (#12ate-)
0,000 (=1.84-)
-0.002 (+405.2))
0.001 (#
cI-MAAE
0.209 (144266))
0,247 (2AJ2)))
0.008 (2:49994)
0,218 (1.271))
0.148 (19225)3)
JMVAE
0.220
0.157 (=499-2)
0,191 (#221e2)
0,212 (11226-2)
0.143 (:2.441)
cIIMVAE
0.255
(1434e-)
0.149 (21355-5)
0,226 (1774e-2
0.202 (1:12-))
0.176 (16226-3)
MMVAE
0.117 (12Ste-))
0.094 (1722163)
0,153 (2114722)
cI-MMVAE
0.206
(1:65--2)
0.136 (21)91))
0,251 (12.092)
20% of data used
cC-MMVAE
0.226
(+469)-2)
0,188 (=2m4-2)
0.273 (=9676-3)
MVAE
0.091
(#266e-2)
0.008 (:488e4))
0,005 (#22003)
0.020 (#875-3)
0.009 (a7.77-2)
c1-MVAE
0.132 (1431e-)
0.192 (139902)
.0.002 (1:56)-1)
0.162 (17me))
0.081 (1:43=3)
IMVAE
0.127
(14776-3)
0,118 (178263)
0,154 (143463)
0,181 (212662)
0.139 (21.394))
eI-JMVAE
0.269 (#126-))
0.134 (=422-4)
0,210 (#2255-2)
0.192 (=144c-)
0.168 (#106)))
4.4 DATA EFFICIENCY Finding:
Contrastive learning on 20% of data matches baseline models on full data.
We plot the quantitative performance of MMVAE with and without contrastive learning against the percentage of the original dataset used, as seen in Figure 6) We observe that the performance of con- trastive MMVAE with the IWAE (cI-MMVAE: red) and CUBO estimators (cC-MMVAE, yellow) are consistently better than the baselines (MMVAE, blue), and that baseline performance using all related data is matched by the contrastive MMVAE using just 10-20% of data. The partial datasets used here are constructed by first taking y% of each unimodal dataset, then pairing to create multimodal datasets @8 reproduce results generated from using 100% of the data in MNIST-SVHN and CUB (Tables aad 2 using only 20% of the original multimodal datasets TTbless I and 2 (bottom)). Comparing results between top vs. bottom in Tables I and ( shows that this finding holds across the models, on both MNIST-SVHN and CUB datasets. This shows that the efficiency gains from a contrastive approach is invariant to VAE type, data, and metrics used, underscoring its effectivvness.
it contains the requisite amount of "related" samples. In addition, we
4.5 LABEL PROPAGATION
Finding: Generative models learned contrastively are good predictors of "relatedness". enabling label propagation and matching baseline performance on full datasets, using only 10% of data.
7

---

Published as a conference paper at ICLR 2021
LOWET AEOURC
MEV WINN
MED colm o4tr
MED DHEPIE
(a) Cross coherence
(b) Joint coherence
Latent accuracy
Figure Performance MMVAE, cI-MMVAE and cC-MMVAE using n% MNIST-SVHN
Here, show that contrastive framework encourages larger discrepancy between the PMI of related vs. unrelated data, as set out in hyppthhhsis 3.1, allowing one to first train the model on a small subset of related data, and subsequently construct classifier using PMI that identifies related samples the remaining data. We now introduce our pipeline for label propagation in details.
Fr
170
S
S
F;
F
Sm
F Fm
F:
(U)
(2)
Mixed data
Related daca (ycal)
Related cata (predicted)
Uniclaned data (raa)
Unielated data (predicted)
Figure 7: Pipeline of label propagation
Pipeline showing in Figure 7 we first construct a full dataset by randomly matching instances in MNIST and SVHN, and denote the related pairs by F, (full, related). We further assume access to only n% of Fr, denoted as S, (small, related), and denote the rest as Fm containing mix of related and unrelated pairs Next, train generative model g S,.. To find a relatedness threshold, we construct a small, mixed dataset S. by randomly matching samples across modalities in S,,. Given relatedness ground-truth for Sm, logpe. Pe. ( (y) for all pairs (æ, y) in Sn and estimate an optimal threshold. This threshold can now be applied to the full, mixed dataset Fm identify related pairs giving new related dataset Fr. which can be used to further improve the performance of the generative model
compute the PMI I(xx y) log pe (z. y)
Results In Figure 8 (a-e), plot the performance of baseline MMVAE (blue) and contrastive MMVAE with IWAE (cI-MMVAE, red) and CUBO (cC-MMVAE, yellow) estimators for term 2 of Eq 4 trained with (solid curves) and without (dotted curves) label propagation. Here, the x-axis is the proportion in size of S, to F,. i.e. the percentage of related data used to pretrain the generative model before label propagation. We compare these results to MMVAE trained on all related data (cyan horizontal line) as a "best case scenario" of these training regimes.
Clearly, label propagation using contrastive model with IWAE estimator is helpful, and in general improvement is greater when less data available: Figure 8 also shows that when S, is 10% of F,, CI-MMVAE is competitive with the performance of baseline MMVAE trained on F,.
For baselines MMVAE and cC-MMVAE however, label propagation hurts performance matter the size of S,. as shown by the blue and yellow curves in Figure 8 (a-e). The advantages of cI- MMVAE is further demonstrated in Figure 8f] where compute the precision, recall, and Fi score of relatedness prediction on F/m> for models trained on 10% of all related data. We also compare to a simple label- propalaaooagttion bascline, where the relatedness of F., is predicted using siamese network (Hadsell et al. 2006) trained on the same 10% dataset. Notably, while the Siamese baseline in Figure is competitive predictor of relatedness, cI-MMVAE amongst all four, making it the most reliable indicator of relatedness. Beyond that, note that with the contrastive MMVAE, relatedness can predicted without additional training and only requires simple threshold computation directly computed using the generative model. The fact that the CI-MMVAE's relatedness- prediction performance is the only one that matches the Siamese baseline strongly supports the view that the contrastive loss encourages generative models utilise and better learn the relatedness between multimodal pairs; in addition, the poor performance cC-MMVAE shows that the advantages of having bounded estimation the contrastive objective by using
the highest F1 score
sereeddness-peedcttioo
8

---

Published as a conference paper at ICLR 2021
LITEE MOLEOO
HANE
cne
co COMME
(a) Cross coherence, S=M
(b) Cross coherence, M=
(c) Latent accuracy, MNIST
Methods
Precision (16
Recall (5)
Fi
MINVAE d.MMVAE
36.13
11:20
0.195
68.20
0.312
Peesentage 1RA46//
Percentage dets (RA11/70
(d) Latent accuracy, SVHN
(e) Joint coherence
(f) Precision, recall, Fi score
Figure 8: Models with and without label propagation using MMVAE, cL-MMVAE and eC-MMVAE. upper-bound for ② is overshadowed by the high bias of CUBO, and that one may benefit more from choosing low variance lower-bound like IWAF.
CONCLUSION
We introduced contrastive-style objective for multimodal VAE, aiming at reducing the amount of multimodal data needed by exploiting the distinction between "related" and "unrelated" multimodal pairs. We showed that this objective improves multimodal training, drastically reduce the amount of multimodal data needed, and establishes strong sense of "relatedness" for the generative model. These findings hold true across a multitude of datasets, models and metrics. The positive results of our method indicates that it is beneficial to utilise the relatedness information when training multimodal data, which has been largely ignored in previous work. While we propose to utilise random variable in the graphical
implicitly through contrastive loss, one may consider relatedness model and see if explicit dependency on relatedness can be useful. It is also possible to extend this idea Generative adversarial networks (GANs), by employing an additional discriminator that evaluates relatedness between generations across modalities. We will leave these interesting directions be explored by future work.
ACKNOWLEDGEMENTS
YS and PHST were supported by the Royal Academy of Engineering under the Research Chair and Senior Research Fellowships scheme, EPSRC/MURI grant EP/N019474/1 and FiveAL. YS was additionally supported by Remarkdip through their PhD Scholarship Programme. BP is supported by the Alan Turing Institute under the EPSRC grant 1012510129/1. Special thanks to Elise van der Pol for helpful discussions contrastive learning.

---

Published as a conference paper at ICLR 2021
REFERENCES
A. Blum and S. Chawla. Learning from labeled and unlabeled data using graph mincuts. In C. E. Brodley and
A. P. Danyluk, editors, Proceedings of the Eighteenth International Conference Machine Learning (ICML 2001), Williams College, Williamstown, MA. USA, June 28 July 2001, pages 19-26. Morgan Kaufmann. 2001.
A. Blum and T. Mitchell. Combining labeled and unlabeled data with co-training. In Proceedings of the eleventh annual conference on Computational learning theory, pages 92-100, 1998.
Y. Burda, R. B. Grosse, and R. Salakhutdinov. Importance weighted autoencoders. In Y. Bengio and Y. LeCun, editors, 4th International Conference Learning Representatiins, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL UtRpattpp//arxii.co
abs/ 1509. 00519
M. C. Burkhart and Shan, Deep low-density separation for semi-supervised classification. In International Conference on Computational Science, pages 297-311, 2020.
T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton, A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference Machine Learning. ICMI, 2020, 13-18 July 2020, Virmaal Event, volume 119 of Proceedings of Machine Learning Research, pages 1597-16077 PMLR, 2020. URL
wtt.:/iingngg...sssss...eeeeeeee press/ peapess/vvll9. v119/chene// .html
A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society series b-methodological, 39(1):122, 1977.
A. B. Dieng, D. Tran, R. Ranganath, J. W. Paisley, and D. M. Blei. Variational inference via X upper bound minimization. In st Annual Conference on Neural Information Processing Systems, NIPS 20/7, pages 2732-2741, 2017.
W. Guo, J. Wang, and S. Wang. Deep multimodal representation learning: A survey, IEEE Access, 7:63373- 63394, 2019.
R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '06), volume 2, pages 2006.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Vegas, NV. USA, June 27-30, 2016, pages 770-778. IEEE Computer Society, 2016. doi: 10.1109/CVPR.201699
[tt/hhtpp://bb.. org/10
1109/CVPR. 2016.90
K. He, H. Fan. Y. Wu. S. Xie. and R. B. Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA. USA, June 19,,, 2020, pages 9726 9735. IEEE, 2020, doi: 10.1109CVVR42200.2220000 URL
https://dci. crg/ 10. 1109/11024220000000000 2020. CC975
0. J. Hénaff. Data-efficient image recognition with contrastive predictive coding. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings Machine Learning Research, pages 4182 4192. PMLR, 2020. URL http:://
proceedings.co mlr..
wtt.://.........s/si111/...///// wwtpppswwse..peeet.ptpppppps wttes////11///nnonf22... htm]
R. D. Hjelm, A. Fedorov, S. Marchilna K. Grewal, P. Bachman. A. Trischler, and Y. Bengio. Learning deep representations by mutual information estimation and maximization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https::/cpentee
net forum? werummiddeee wim.miidddllrrr 1r OccKX
C.-C. Hsu, C.-W. Lin, W.-T. Su, and G. Cheung, Sigan:
Siamese generative adversarial network for identity- preserving face hallucination. IEEE Transactions on Image Processing, 28(12):6225 6236, 2019.
Joachims. Transductive inference for text classification using support vector machines. In ICML '99 Proceedings of the Sixteenth International Conference on Machine Learning, pages 200-209, 1999.
D. Massiceti, P. K. Dokania, N. Siddharth, and P. H. Torr. Visual dialogue without vision or dialogue. In NewrlPS Workshop on Critiquing and Correcting Trends in Machine Learning.
D. J. Miller and H. S. Uyar. A mixture of experts classifier with learning based on both labelled and unlabelled data. In Advances in Neural Information Processing Systems 9, pages 571-577, 1996.
10

---

Published as a conference paper at ICLR 2021
A. Mnih and K. Kavukcuoglu. Learning word embeddings efficiently with noise-contrastive estima- tion. C. J. C. Burges, L. Bottou, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Process- ing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe Nevada, United States, pages 2265-2273, 2013. BRRhhttps: pttp:://ppoccccngg...ooo db2b41 8215622f (tt.ee/ee0011110000090010 (0w.06000000000
www.eeidinnnnnuurrip...ee neur ips. http:/aapr/211//aaaaaa// Abstract........................ .html
K. Nigam, A. K. McCallum, S. Thrun, and T. Mitchell. Text classification from labeled and unlabeled documents using em. Machine Learning, 39(2):003-134, 2000.
S. E. Reed, 7., Akata, H. Lee, and B. Schiele. Learning deep representations of tineggrinee visual descriptions. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV. USA. June 27-30, 2016, pages 49-58. IEEE Computer Society, 2016. doi: 10.1.00/CV..2011.11... URL htsps
//doi.org/lu.1 201.
Shi, S. Narayanaswamy, B. Paige, and P. H. Torr. Variational mixture-of autoencoders for multi- modal deep generative models. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché- Alché-Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrlPS 2019, December 8-14, 2019, Vancouver, BC.
Canada, pages 15692-15703, 2019. URL httttts://pocceeeeee igg .aw.inurinnnnnnnnnmmmmm ps. cc/paapr/2201) 563 httm
H. O. Song. Y. Xiang, S. Jegelka, and S. Savarese. Deep metric learning via lifted structured feature embedding. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27 2016. pages 4004 4012. IEEE Computer Society, 2016, doi: 10. 10.1109/CVPR.2001444 URL https:77dee org.
434
M. Suzuki, K. Nakayama, and Y. Matsuo. Joint multimodal learning with deep generative models. In Interna- tional Conference on Learning Representations Workshop.
Y. H. Tsai, P. P. Liang. A. Zadeh, L. Morency, and R. Salakhutdinov. Learning factorized multimodal representa- tions. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, 2019, URL https: | /ttp::::///peeee///.eee net/ wwwpe//ooormmdoreeeeeeeeeee 9KX
A. van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1kkvv/800.000 887400378... 2018.
R. Vodantam, I. Fischer, J. Huang, and K. Murphy. Generative models of visually grounded imagination. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canuda, April 30- May 2018, Conference Track Proceedings. OpenReview.net, 2018. htRt:ttt::. wwwp////ppp/eeee..e/o/o/e. n Forum? Ad
T. Wang and P. Isola. Understanding contrastive representation learning through alignment and uniformiity on the hypersphere. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Viriual Event, volume 119 of Proceedings of Machine Learning Research, pages 9929-9939. PMLR, 2020. URLhhtt://ppoceee
ngs .mlz. www.ss/sssssee.....ee vv1 1119 9/ ////aang22 20k. html
K. Q. Weinberger, J. Blitzer, and L. K. Saul. Distance metric learning for large margin nearest neighbor classification. In Advances in Neural Information Processing Systems 18 [Neural Informa- tion Processing Systems, NIPS 2005, December 5-8, 2005, Vancouver, British Columbia, Conada], pages 1473-1480, 2005. URL 00099226e88111669666229 111.////991111111111111 /ww.6115bbb--.. Aww.bbb..bbaaattt......
ca/ wwt.nnuis....ppppeeeeeee
Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010000 California Institute of Technology, a.
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology, b.
M. Wu and N. D. Goodman. Multimodal generative models for scalable weakly-supervised leam- ing. S. Bengio. H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi. and R. Gar- nett, editors. Advances in Neural Information Processing Systems 31: Annual Conference on Neu- ral Information Processing Systems 2018, NewrIPS 2018 December 3-8, 2018, Montréal, Canada, pages 1580055900 2018. URL -bstraccc.--btttt......t
G. Yildirim, N. Jetchey, and U. Bergmann. Unlaheled disentangling of gans with guided siamese networks. 2018.
11

---

Published as a conference paper at ICLR 2021
L Yildirim. From perception conception: Learning multisensory representations. 2014.
C. Zhang, Z. Yang. X. He, and L. Deng. Multimodal intelligence: Representation learning. information fusion, and applications. IEEE Journal of Selected Topics in Signal Processing, pages 1-1. 2020.
D. Zhou, O. Bousquet, T. N. Lal, J. Weston, and B. Schölkopf. Learning with local and global consistency. In S. Thrun, L. Saul, and B. Schölkopf, editors, Advances Neural Information Processing Systems 16 [Neural Information Processing Systems, NIPS 2003, December 8-13, 2003, Vancouver and Whistler, British Columbia, Canada pages 321 328. MIT Press, 2003, URL ht tps /ptp://doccccc..epeeeo neurips....... cc/ .html
paper 2003 hash/ 8 24
Z. Zhou and M. Li. Tri-training: exploiting unlabeled data using three classifiers. IEEE Transactions on Knowledge and Data Engineering, (7(11)1529-1541, 2005.
Z.-H. Zhou and M. Li. Semi-supervised learning by disagreement. Knowledge and Information Systems, 24(3): 415 009,4499,00 2010.
X. Zhu, Z. Ghahramani, and J. D. Lafferty. Semi- supervisee learning using gaussian fields and harmonic functions. In T. Faweett and N. Mishra, editors, Machine Learning, Proceedings of the Twentieth International Conference (ICML 2003), August 21-24, 2003, Washington, DC, USA, pages 912 919. AAAI Press, 2003. URL /wwww//www.////.////////////ww org/ Libra ry/ ICMLI//ILL///22
cm
php
12

---

Published as a conference paper at ICLR 2021
Appendix:
A
FROM POINTWISE MUTUAL INFORMATION TO JOINT MARGINAL LIKELIHOOD
In this section, we show that for the purpose of utilising the relatedness between mutlimodal pairs, maximising the difference between pointwise mutual information between related points æ, y and unrelated points æ, y' is equivalent to maximising the difference between their log joint marginal likelihoods.
To see this, we can expand ((x, y) [xx, 3/) as
(x, y) [xx y')
[log pe (x, y) log (z) log po)) [log pe (x, 3/) log (o()) log (e log pe (x, v) log pel ar, y²) log pe (y') log pe (
In 66) the PMI difference is decomposed
terms: 1 the difference between joint marginal likelihoods and 2 the difference between marginal likelihoods of different instances of g-s. It is clear that since 2 involves only one modality and only accounts for the difference in values between and y', it is not relevant to the relatedness of æ.
Therefore, for the purpose of utilising relatedness information, we only need to maximise (xx y) [xx y') through maximising term ① i.e. the difference between joint marginal likelihoods.
B CONNECTION OF FINAL OBJECTIVE TO POINTWISE MUTUAL INFORMATION
Here, we show that minimising the objective in maximises the PMI between æ and y::
C(x, y) = -y --yl-ygg pe (z, 8) + 2
log Σ
pe(x', y) log Σ pe yo)
x'E
y'EY
pe pe (a. y)
- 2)
log pe (x., y)
log
Ey'er (e v²) Ex'ex (e(
pe (æ, y)
~ -77 2 (x,
log
2
pel (xr pe (1)
((x.y)
We see in 7 that minimising L can be decomposed to maximising both the joint marginal likelihood Pe (e 3) and an approximation PMI [xx, y). Note that since y > 1, we can be sure that the joint marginal likelihood weighting 7 z is non-negative.
13

---

Published as a conference paper at ICLR 2021
C GENERALISATION TO M > 2 MODALITIES
In this section we show how the contrastive loss generalise to cases where number of modalities considered M is greater than 2.
Given observations from M modalities = X2, _m. X where Xm denotes (1)
unimodal dataset of modalit of size Nm. i.e. X {arm ji=-- Similar to assymetrical contrastive loss for any observation am from modality m, where negative samples are taken for all (M 1) other modalities:
write the
M
N
Lo(am Di. log pe (2 1 log
Σ
Pe
(a
(3)
1:(d 1) (
(8)
d=1 1=1 dem
We therefore rewrite 3 as:
La(ar(:ss)
Σ (ar Dmn)) M
1
M
log pe (#i:s))
1
log
pe
(0) lidd 10
(10)
M
j
where N is the number of negative samples, all log (#I: are approximated by the following joint ELBO for M modalities:
log (e x1:M)
log (F 2 Ez (2
EELOO
(II)
go x1:M)
While the above gives the generalisation of (3 we note that the number of times where ELBO needs to be evaluated in (10) is O(M²N), making it difficult to implement this objective in practice, especially on more complicated datasets. We therefore propose a simplified version of the objective, where we estimate the second term of (1O) with N sets of random samples from all modalities. Specifically, we can precompute the following M N random index matrix
j1
f12
JIN
J21
Je2
J2N
J
(12)
jim
JM2
jMN_
where each entry of J is random integer taken from range [1, Nm)- We then replace the second term of (10) random samples selected by the indices in J, giving us
( x - -log pe (m +
log
(Jan) (J2n) pe (ax 2t a
M
(Jasn))
(13)
==1
The number of times ELBO needs to be computed is now O(N), and is no longer relevant to the number modalities M.
We can now also generalise the final objective in to M modalities:
L( = (E (1)
[logsumexp ELBO(x (J1n), (J2m)
22 ; ... FM (hann)))
(14)
nEN
14

---

Published as a conference paper at ICLR 2021
D THE INEFFECTIVENESS OF TRAINING WITH LO ONLY
We demonstrate why training with the contrastive loss proposed in (3) ineffective, and why additional ELBO term needed for the final objective. As show in Figure 9 when training with Le only, while the contrastive loss (green) quickly drops zero, both term ! and in ( also reduces drastically. This means the joint marginal likelihood of any generation log pe (z, y) is small regardless the relatedness of (z, y)
44800
-5000
-2000
Lc
-5400
-5600
-6000
100
150
200
250
Iterations
Figure 9: First 300 iterations of training using contrastive loss Le only.
In comparison, we also plot the training curve for model trained on the final objective in which upweights term ! in (3) by y- We see in/uurure that by setting ? 2, the joint marginal likelihood (yellow and blue curve) improves during training, while Le (green curve) gets minimised.
200
-3000
-3000
-3750
-4000
Le
-4250
45500
-5000
100
200
250
Iterations
Figure 10: First 300 iterations of training with final loss L, where 7
15

---

Published as a conference paper at ICLR 2021
E ABLATION STUDY OF
In $ we specified that y needs to be greater than to offset the negative effect minimising ELBO through term in Here, we study the effect of y in details.
Figure 1 compares latent accuracy, cross coherence and joint coherence of MMVAE on MNIST- SVHN dataset trained on different values of y. Note that here we only consider cases where Y 2 since the minimum value of y is 1. In this case, the loss reduces to the original contrastive objective
few interesting observations from the plot using the contrastive loss only, and as we showed an ineffective objective for generative model learning. This is verified again in Figure 11 when y 1, both coherences and latent accuracies take on extremely low values; interestingly, there is significant boost of performance across all metrics by simply increasing the value of from 1.1; after that, as the value of y increases, performance on most metrics decreases monotonically (joint coherence being the only exception), and eventually converges baseline MMVAE (dotted lines in Figure IT). This unsurprising, since the final objective in reduces to the original joint ELBO approaches infinity.
follows: First, when 7 1, the model is trained
100
AccuracylCoherence (%)
40
MNIST Latent
SVHN Latent
Cross coherence, M =
Cross coherence, S
Joint coherence
Figure 11: Performance different metrics for different values of Dotted lines represents the performance of baseline MMVAE.
seems to suggest that 1.1 is the optimal value for hyperparameter 7, however close inspection of the qualitative generative results shows that this might not be the case. See FeeuFe 12 for comparison of the model's generation between MMVAE models trained (from left right) y y= 1.1, = 2 and 7 = +oo (i.c. original MMVAE). Although 7 = 1.1 yields model with high clearly from the left-most column of Figure 12 that the generation of
Figure 11
coherence scores, the model seems deprecated, especially for the SVHN modality, where the backgrounds of model's generation appear to be unnaturally spotty and deformed. This problem is mitigated by increasing - as shown in Figure 11, the image generation quality of 7 = 2 (middle column) is not visibly different from that of +o (right column).
To verify this observation, also compute the marginal log likelihood log pe (x, y) to quantify the quality of generations. We compute this for all ys considered in Figure 11 and take the average the entire test set. From the results in Figure 13] likelihood between y = 1.1 to = 1. This gain in image generation quality then slows down as further increases, and as all other metrics converges to the original MMVAE model.
significant increase of the log
16

---

Published as a conference paper at ICLR 2021
7 1.1
Y 2
Y
+
5
C 0
O i 4 : 8
O
1 3 2 9 3
4
6
+
0
1
0
4
8
0
5
4
6
9
2
1
0
3
5
5
5 3
0
7
2
3
6
3
8
41
O
6 3
2 7 4 9
5 3 8 6 6
9
7
3
9
3
5 6
1
0
7
3
6
2
9
9
2
3
7
3
2
5
5
9
8
9
6
6
9
9
a
3
3
J
0
0
5
5
5
7
5
5
7
5
6
5
4
1 41
8
6 5
8
5
3
4
5
7
8
4
9
8
5
0
1
3
2
3
5
3
5
9
5
6
3
7
7
6
6
3
7
1
8
7
2
3
2
0
6
3
5
9
R
3
8
(a) Joint generation, MNIST
(b) Joint generation, SVHN
3
2
8
3
3
/
5
3
5
S 2 4
3
9
8
4
3
7
P
3
3
2
5
5
3
/
5
3
5
5 2 4
3
G
8
4
3
7
7
3
(c) Reconstruction, MNIST
3
2
8
2
3
/
5
3
9
8
4
3
7
3
(d) Reconstruction, SVHN
2
3
A5
3
2
2
8
2
9
3
5
3
5 /
8
5
4
7
7
(e) Cross generation, S-*M
2
3
N 3
38
(f) Cross generation, M-
Figure 12: Generations of MMVAE model trained using the final contrastive objective, with (from left to right) = 1.1, and +00. Note in (c), (d), (e), (f), the top rows are the inputs and the bottom rows are their
corresponding reconstruction/cross generation.
2300
Likallhaoots
+200
-3200
Likkhoods
Figure Performance on different metries for different values of Dotted lines represents performance baseline MMVAE.
17

---

Published as a conference paper at ICLR 2021
F ARCHITECTURE
We use architectures listed in Table 3 for the unimodal encoder and decoder for MMVAE, MVAR and JMVAE. For JMVAE we use extra joint encoder, the architecture of which is described in Table 4
Encoder
Decoder
Input e R1a25125
Input RR4
PC. 400 ReLU
FC, 400 ReLU
FC. L. L
FC, x 28 28 Sigmoid
(a) MNIST dataset
Encoder
Input E R35552252
4x4 conv. 32 stride pad ReLU
4x4 conv. 64 stride 2 pad & ReLU
4x4 conv. 128 stride 2 pad & ReLU
4x4 conv. L stride I pad 0, 4x4 conv. L stride I pad 0
Decoder
Input E R.
4x4 upconv. 128 stride I pad 0 & Rel.U
4x4 upconv. 64 stride pad & ReLU
4x4 upconv, stride pad & ReLU
4x4 upconv, stride 2 pad Sigmoid
(b) SVHN dataset.
Encoder
Decoder
Input 2048
Input e i
FC. 1024 ELU
FC, 256 ELU FC, 512 ELU FC, 1024 ELU
FC, 512 ELU
FC, 256 ELU
FC. L. FC.
FC. 2048
(c) CUB image dataset.
Encoder
Input 15500
Word Emb. 128
4x4 conv. 32 stride 2 pad & ButchNorm2d & ReLU
4x4 conv. 64 stride 2 pad & BatchNorm2d & ReLU
4x4 conv. 128 stride pad & BatchNorm2d & ReLU
1x4 conv. 256 stride 1x2 pad 0x1 & BatchNorm2d & ReLU
1x4 conv. 512 stride 1x2 pad 0x1 BatchNorm2i & ReLU
4x4 L stride pad 0, 4x4 L stride pad
Decoder
Input RC
4x4 upconv. 512 stride pad ReLU
1x4 upconv. 256 stride 1x2 pad 0x1 & BatchNorm2d & ReLU
1x4 upconv. 128 stride 1x2 pad 0x & BatchNorm2d & ReLU
4x4 upconv. 64 stride 2 pad & BatchNorm2d & ReLU
4x4 upconv. 32 stride 2 pad 1 Norm22 & ReLU
4x4 upconv. stride pad & ReLU
Word Emh.? T 1590
(d) CUB-Langune dataset.
Table 3: Unimodal encoder decoder architectures.
18

---

Published as a conference paper at ICLR 2021
Encoder
Input E R33332664
4x4 cony. 32 stride 2 ped & ReLU
4x4 cony. 64 stride 2 pad & ReLU
4x4 cony. 128 stride 2 pad
ReLU
1x4 cony. stride pad Ox1 ReLU
4x4 conv. stride pad 4x4 conv. stride I pad
(a) MNIST-SVHN dataset.
Encoder
Input Rla322102
4x4 conv. 32 stride pad & BatchNorm2d & ReL.U
4x4 conv. 64 stride pad & BatchNorm2d & ReLU
conv. 128 stride pad 1 BatchNorm2d ReLU
RatchNorm2id ReLUU 1x4 512 stride 1x2 0x1 BatchNorm2d ReLU
conv. 256 stride 1x2 pad
4x4 conv. L. stride pad 0 4x6 conv. L stride pad 0
(b) CUB Image-Caption dataset.
Table 4: Joint encoder architectures.
19

---

Published as a conference paper at ICLR 2021
G
QUALITATIVE RESULTS ON MNIST-SVHN
G.1 GENERATIVE RESULTS & MARGINAL LIKELIHOODS ON MNIST-SVHN
G.1.1 MMVAE
MMVAE
cI-MMVAE
CC-MMVAE
0
4
0
5
7
1
5
3
4
9
7
0
2
3
6
3
8
7
6
1
9
9
5
7 8 1
1
0
7
3
6
9
7
6
0
8
6
3
4
e 0
5
0
9
8
4
6
6
9
9
3
8
2
3
1
8
8
1
5
6
3
4
8
0
4
1
7
7
8
4
5
0
9
0
8
6
6
6
7
9
9
1
6
9
9
7
6
0
0
0
0
0
3
5
R
3
8
5
6
2
3
6
5
9
(a) Joint generation, MNIST
8
(b) Joint generation, SVHN
8
4
7
P
7
9
4
2
0
2 2
9 7
3
8
4
3
7
3
4
2
0
(c) Reconstruction, MNIST
38
0
89
6
105
19
1
15
M4
(d) Reconstruction, SVHN
8
6
50S
19
8
5
4
7 7
9
9
5
8
2
9
(c) Cross generation, S-MM
3 9
8
4
7 P
2
4
4
2.
0
2
9
(f) Cross generation, M-
Figure 14: Generations of MMVAE model, from left to right are original model (MMVAE), contrastive loss with IWAE estimator (cI-MMVAE) and contrastive loss with CUBO estimator (cC-MMVAE))
log p(mm, vn)
log
Dm, Fn)
Dog((Em.
3m
log log xxn)
MMVAE
-1879.00
-388.59
-388.59
-1618.52
MNIST,
CI-MMVAE
11904.15
-385.18
1885.18
- -1620.77
SVHN
CC-MMVAE
-1924.20
-391.88
-391.84
-1619.34
MMVAE
1879.00
-1472.44
-1472.45
-33.66
SVHN. MNIST
cI-MMVAE
-1904.15
-1491.50
-1491.55
- 444.28
cC-MMVAE
-1924.20
-1490.56
-1494.75
-28.29
Table 5: Evaluating log likelihoods using original model (MMVAF), contrastive loss with IWAE estimator (cI-MMVAF) and contrastive loss with CUBO estimator (cC-MMVAF). Likelihoods are estimated with IWAE estimator using 1000 samples.
20

---

Published as a conference paper at ICLR 2021
G.1.2 МVAE
MVAE
CI-MVAE
cC-MVAE
5
6
7 0
1
9
4
7
2
7
3
3 4 9 # 4 4 5
8 6 3 5 4 5 3
2
6
7 7
4 7
7 0 5 6 3
6 9
3
5
3
4
0 7 9 5
1
7
4
7 2
7
3
4
8
7
2
7
-2
5
1
1
I
5
3
4
9
3
6
4
9
1
5
5
5. $ 5
7
J
5
2
1
5
0 4
0 8
3
5
5
4
2
4
7
5
2
1
1
4
5
1
7
9
6
7
2
J
3
8
(a) Joint generation, MNIST
(b) Joint generation, SVHN
G
5
4
7
8
4
0
3
S
7
8
2
4
0
5
(c) Reconstruction, MNIST
17
61
(d) Reconstruction, SVHN
17
17
6
3
S
6
4
7
1
9
8
4
O
S
(c) Cross generation, S S-MM
5
5
6
4
7
4
0
1
(f) Cross generation, M-
Figure 15: Generations of MVAE model, from left to right original model (MVAE), contrastive loss with IWAE estimator (cI-MVAB), contrastive loss with CUBO estimator (cC-MVAE).
log p (mm; vn.)
log pm#m.
3m, Fn.)
log p
3mm)
log pmmm
MVAE
-404.43
440.44
388.27
-1847.85
MNIST,
cI-MVAE
-406.85
-00.22
388.26
1876.95
SVHN
CC-MVAE
-405.24
-432.96
-388.26
11889.35
MVAE
404.43
-1518.00
-1488.21
440.32
cI-MVAE
406.85
- -1529.32
-1498.73
443.04
MNIST
cC-MVAE 405.24
-1520.23
-1499.47
-44.00
Table Evaluating log likelihoods using original model (MVAE), contrastive loss with IWAE estimator (cl- MVAE) and contrastive loss with CUBO estimator (cC-MVAE). Likelihoods are estimated with IWAE estimator
using 1000 samples.
21

---

Published as a conference paper at ICLR 2021
G.1.3
JMVAE
JMVAE
CI-JMVAE
CC-JMVAE
6
4
J
7
3
7
7
4
8
4
9
3
6
7
4
c
9
6
4
0
9
3
7
0
9
f
1
9
9
O
9
0
7
4
6
6
4
1
>
6
6
9
4
7
7
7
8
7
1
5
7
8
3
1
7
6
0
4
1
7
9
2
7
3
3
9
4
7
1
6
7
9
6
4
5
4
7
J
7
9
6
(a) Joint generation, MNIST
(b) Joint generation, SVHN
3
6 8
6
2 2
9 5
7
8
3
9
9
3
6
8
6
5
1
9
9
3
(c) Reconstruction, MNIST
5
(d) Reconstruction, SVHN
Figure 16: Generations of JMVAE model, from left to right are original model (JMVAE), contrastive loss with IWAE estimator (cl-JMVAB), contrastive loss with CUBO estimator (cC-JMVAE).
log p ((0m; Dn)
log
[m,
JMVAE
-515.44
-497.95
MNIST. SVII
cI-JMVAE
-518.56
-511 121
cC-JMVAE
-534.26
-510.90
JMVAE
515.44
1515.73
SVHN
eI-JMVA
-51.556
-1529.44
MNIST
cC-JMVAE -334.26
-1614.78
Table 7: Evaluating log likelihoods using original model (JMVAE), contrastive loss with IWAE estimator (cI-JMVAB) and contrastive loss with CUBO estimator (cC-IMVAB). Likelihoods are estimated with IWAE estimator using 1000 samples.
22

---

Published as a conference paper at ICLR 2021
G.2 GENERATION DIVERSITY
To further demonstrate that the improvements from our contrastive objective did not come at price of sacrifising generation diversity, in Figure 17 we show histograms of the number of examples generated for each class with samples from prior. Similar to how we compute joint coherence, the class label of generated images are determined using classifiers trained on the original MNIST and SVHN dataset.
2700
(a) MMVAE
(b) cL-MMVAE
(c) cC-MMVAE
(d) MVAE
cl-VAE
cfCC-MVAE
2000
(g) JMVAR
(h) cI-JMVAE
cC-IMVAR
Figure 17: Number of examples generated for each of class during joint generation.
We can see that the contrastive models (cI-* and cl-+) are capable of generating examples from different classes, and for MMVAE and MVAE the histogram of contrastive models are more uniform than the original models (less variance between class counts).
23

---

Published as a conference paper at ICLR 2021
H QUALITATIVE RESULTS ON CUB
The generative results of MMVAE, CI-MMVAE and cC-MMVAE on CUB Image-Caption dataset are as shown in Figure 18 Figure 19 and Figure 20 Note that for the generation in the vision modality, we reconstruct and generate features from ResNet101 and perform nearest neighbour search in all the features in train set to showcase our generation results.
Image Image
Sentence + Image
[DATA] wex sttall stzdd hird that has crmeiny with short pottiaed 1IIl.
bluch
Sentence
[DATAA] mex serill bind with white teity and wing
pounty
aboct look. -oous
sboet bvak,
[DATTA] sas trown with while
pointy
Image Sentence
brak cnoss
brak -001>
longs pobity
[RBOON] this and har stort bealt coeato
wange
<coo>>
belly
[[N]CUN] wings that are grey and has yeauow
[RBONN) this back bard black
and book.
[HUCON]
ar while beily short
(RECON] wade this biid haid with heng whice neck and orange billl -eeeo
Joint generation from prior samples:
IGENI Nird
ICEN] small Sind with bill and trown
[PRN] this hind belly
yellow yellinn
Figure 18: Qualitative results of MMVAE on CUB Image-Caption dataset, including reconstruction (vision + vision, language language), cross generation (vision -> language, language - vision) and joint generation from prior samples.
Image
Image
Sentence + Image
[GTN]] mep:
that has
trerad and beity with boown spola, the bouk puinted
[GEN] wea Lake staall
bownn over with shart beak moosie
Sentence
[ritN] -us tiik himt has odtage that aze yelow and has black theoot. <000c-
[3ATA]] smdll with white felly grey
thort bosk <005>
[SEXXN]
Munt beak. soull with whsle befly and
Image
Sentence
long, potnty
boak, K000
[190XNN]
back wittle and hax long, potrty
[FFN] u this iint has Nivi
cmam, long and black caceo.
DAT
witgo that are groy and las a yetoww till
[3800]] ->> this 20rd baz wtags that are groy and bao a yollow
(CEN) with thes particatur Mird huo while beily and wings that are beews. wow.
hav very [XXXXNN] --> this hint to white with betty and and shart trssk.
[TATT]] this whilr
(UNN) wao tia tund tan Mixis crwwn and white and and white belly scsss-
Joint generation from prior samples:
lempe
Figure 19: Qualitative results of MMVAE trained with contrastive loss with IWAE estimator CUB Image- Caption dataset, including reconstruction (vision vision, language - language), cross generation (vision language, language -> vision) and joint generation from prior samples.
24

---

Published as a conference paper at ICLR 2021
Image + Image
Sentence Image
[DAI trkk spreiiled
black and white sperteed brovs belty
(DATA) land bully
Sentence
[DATM cound Sestuave
(0007]] --> lhe sind has losly tiad is while with bronn sklen.
(NX/U/) tae
betly yellown and
Image
Sentence
[DA2A] --. lloo aed has wings thdt ure Llux and lus a wiite book.
(RBOON) --> Ca bord had wteg: that are bive and has an ortinge
(RECON] --- thi lird a quey with are usd hae very stort beak vosoe
(DAIS] bied has prey body and whute and grey spockled
[RDOONI mmp the tirt has small bill, and tlwck, and and arew and
[RDOON] biack
[DAIAU lorge and round hind colors of block
[HUCOON] --th a vebite and white hind hae blace becwm black and
Joint generation prior samples:
[[2]]] meo small
[GEN) yetuow Mack with yutkow and -eeet
[GBNN] tars tstt end and widl and krowa emion senvo
yetiow witto uny
vidil bard wiLL shart kmng bwik heak seeako
Figure 20: Qualitative results of MMVAE trained with contrastive loss with CUBO estimator on CUB Image- Caption dataset. including reconstruction (vision -> vision, language - language), cross generation (vision + language, language vision) and joint generation from prior samples.
25


Reviews:
[{"review_id": "sdfoXW7T79", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": " The paper proposes a contrastive objective that (1) minimizes the distance between \"related\" samples while (2) maximizing the distance between randomly paired samples. Existing multimodal VAEs optimize (1) via different multimodal ELBOs. The novelty lies in the optimization of (2) which can further benefit from unimodal samples for which no \"related\" samples of the other modality are available---this can be viewed as a semi-supervised approach for weakly-supervised multimodal data.  For the estimation of (2), the paper experiments with two different estimators, IWAE and CUBO.\n\nThe paper claims three contributions:\n- (C1) improve multimodal learning of existing multimodal VAEs by extending them with objective (2)\n- (C2) improve sample efficiency of multimodal VAEs by extending them with objective (2)\n- (C3) further improve sample efficiency by bootstrapping \"related\" samples from a larger pool of unimodal samples\n\nOverall, the empirical results support the claims of improved sample efficiency, but there might be a potential problem with claim (C1), due to the one-sided selection of metrics used for evaluation.\n\n\n## Strong points\n\nThe paper tackles a practical problem (sample efficiency) with a\nstraightforward solution (leverage unpaired data) that can be easily\nimplemented and seems to work well across a family of models.\n\nStrong empirical results demonstrating improved sample efficiency. For the\nconsidered datasets, the proposed objective improves sample efficiency \nsignificantly, requiring only 20% of the dataset to reach a\nperformance comparable to not using the additional objective. Additional\nresults showing that the sample efficiency can further be improved by a\nprocedure for bootstrapping paired samples from a set of unpaired samples.\nThis is an interesting idea that has been explored in semi-supervised learning,\nbut the present paper applies it to a weakly-supervised, multimodal data.\n\nThe paper nicely connects the objectives used in multimodal VAEs to contrastive\nlearning. It extends the different ELBOs used in multimodal VAEs with an\nadditional objective for minimizing the similarity of randomly paired samples.\n\n\n## Weak points\n\nMy main objection is with regards to claim (C1), which states that the new\nobjective improves multimodal learning. The problem is that the choice of\nmetrics might be too one-sided, measuring only the \"relatedness\" in terms of\nground truth labels, but not the generative quality (of a generative model). As\nsuch, the used metrics (linear classification accuracy; classification accuracy\nof a pretrained classifier on generated samples) ignore the diversity of\ngenerated samples. For example, a model could achieve perfect scores by\ngenerating a single \"canonical\" image of the corresponding class, akin to mode\ncollapse. I understand that generative quality is not the paper's primary goal,\nbut it should not be overlooked, since otherwise, it begs the question of why to\nuse decoders at all and not just apply contrasting in latent space. The\nablations in the appendix indicate a trade-off between the \"relatedness\"\nmetrics and generative quality, suggesting that the proposed objective yields a\nlooser ELBO (Figure 13) and results in a worse generative performance (Figure\n12). If such a trade-off exists, I would recommend presenting it more\ntransparently in the experiments and phrasing the respective claim more\nconservatively.  Further, all previous multimodal VAEs report log-likelihood\nvalues, so it seems natural to provide these numbers, too.\n\nIn the related work section, it is not clear how the authors conclude that contrastive methods are limited to \"specific tasks\" and how the proposed model overcomes this limitation. If the statement refers to the manual\ndesign of a contrastive objective, I would argue that previous work includes\nexperiments showing that learned representations still generalize to other\ndownstream tasks, such as classification and object detection. If the authors\nrefer to generative tasks in particular, this should be stated more clearly.\n\nOverall, it is a good paper with strong empirical results for the sample\nefficiency argument. Yet, I tend towards a weak accept, due to the one-sided\nevaluation. I will be happy to adjust my score if the paper presents the\ngenerative results more transparently in the experiments.\n\n\n## Questions\n\nIn the experiments of Section 4.3, do I understand correctly that when using\ne.g. 20% of the data, you do not use any of the remaining 80% as \"unrelated\"\nsamples for contrasting? Put differently, I understand that you only use\nrandomly paired samples among the 20%-subset for that purpose, correct?\n\nIn the description of the datasets you mention that multiple pairings are used\nto create a dataset of \"related\" samples.  Have you also tried to reduce the\nnumber of pairings instead of taking $n$ percent of the dataset? Does this make\na difference with respect to classification, coherence, or generative\nperformance?\n\n\"Relatedness\" in this work is based on log-likelihoods computed in sample\nspace.  However, contrasting in latent space might provide a more meaningful\nmeasure of \"relatedness\". Have you considered computing the objective (or at\nleast the part that estimates the \"unrelatedness\") using latent representations\ninstead?\n\nIn Section 4.5 it is not quite clear how you estimate the \"optimal threshold\".\nWhat measure is this threshold based on? Do the results in Figure 8 depend \non this threshold and would it be reasonable to show an ablation across\ndifferent threshold values?\n\nIn the paragraph above Equation (2) you mention the margin $m$.  It is not\nclear why $m$ is not used later on. E.g., Is it an additional hyperparameter in\nthe model?\n\n\n## Additional Feedback\n\nIn Hypothesis 3.1., you use the pointwise mutual information (PMI) and state\nthat PMI(x, y) > PMI(x, y'). As far as I understand, PMI is zero when two\noutcomes are independent, but it can be negative when two outcomes are not\nindependent, which would violate the \"uncontroversial assumption\".\nAlternatively, maybe one can use the mutual information between two random\nvariables instead of PMI between outcomes?\n\nThe related work section could benefit from a short paragraph on\nweakly-supervised learning, which seems to be an important theme in the present\npaper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "kMP17hFgPuM", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper tries to minimize the difference of the PMI between related and unrelated pairs of multimodal data but arrives at a very different objective with many approximations. It can be plugged into existing VAE based methods and improve learning performance and data efficiency. My major concern is about the derivation and the connection between motivation and the final objective.\n\nClarity and correctness:\n\nThe presentation of Section 3 should be further clarified. In particular, I wonder about the validness of Hypothesis 3.1. First, it is not clear to me which joint distribution is used in the definition of PMI in the hypothesis? It is not specified and I guess it is the model distribution after training according to the derivation in Appendix A. It is kind of strange because we make a hypothesis about the model after training and optimize the model to reach the hypothesis. \n\nThe derivation is also unclear, according to Eqn. (6), the objective should be the difference between both joint and marginal distributions. I'm not fully convinced by simply ignoring the difference between the marginal ones. I note that the authors mention that the marginal term is not relevant to the relatedness but ignoring it is not optimizing the difference between the PMI, right?\n\nEqn. (6) to Eqn. (2) is also unclear, why we optimize log sum p(negative) instead of optimizing sum log p(negative)? log sum negative is quite like an approximate of log marginal, which seems to contradict the discussion under Eqn. (6). \n\nThe resulting final objective (4) does not obviously connect to the original motivation: maximizing the difference between the PMI values of related and unrelated pairs. I checked Appendix B while the authors claim that it approximately optimizes the joint loglikelihood with an IPM regularization. I have several questions: i) how does this connect to the original motivation? ii) how does the approximation in Eqn. (7) affect training? iii) can we directly optimize Eqn. (7) instead of Eqn. (4)?\n\nThe estimate of the final objective also involves unknown approximation. In particular, the paper does not use CUBO and IWAE properly to form a valid bound of the objective. \n\nWith so many approximations, I can hardly figure out the key factors of the proposed method and say about the technical contribution.\n\nExperiments:\n\nI'm not an expert in multimodal learning. The paper mainly focuses on comparison with VAE based methods. The evaluation metrics are not commonly used but chosen following a related work. The paper claims that the main empirical contribution is its data efficiency, which seems to be verified.\n\n===========\n\n**after rebuttal**\n\nSome of the issues are clarified. I updated my score to 5.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "SyvZxQCei1I", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents a generative model for multimodal learning. The paper maximizes or minimizes the pointwise mutual information between data from two modalities considering a novel random variable relatedness to dictates if data are related or not.  This is realized by casting multimodal learning as max-margin optimization with the contrastive loss for the objective. For the optimization, the paper considers the IWAE estimator. As per the experiments, the paper considers MNIST-SVHN and CUB Image-Captions dataset and perform evaluations across four metrics. Using the experiments, the paper demonstrates that the proposed approach improves multimodal learning, data-efficient learning, and label propagation. \n\n- The paper is well written and easy to follow.\n- The experimental design is praiseworthy as the paper considers different aspects in evaluating multimodal learning via a generative model. The optimization of the proposed approach is also carefully considered. \n- The experimental results across all three tasks demonstrate the efficacy of the proposed approach. \n- One weakness that I find is a lack of comparison against the GAN based models. Can the authors do such a comparison? Or at least discuss why and how GAN based models could be integrated with the current approach? \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "FmrYMYAcnBU", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe paper proposes a contrastive multimodal generative model framework for including unrelated datapoints as well as related datapoints in the learning of the multimodal model. Using such a contrastive formulation, the paper shows that the proposed model outperforms previous work on four desiderata for multimodal generative models. \n\nStrengths\n+ The paper is a novel take on contrastive learning in a probabilistic generative modeling framework, which is interesting\n+ The paper reports extensive experimental results\n\nWeaknesses\n\nWhy not directly use some energy based model with a contrastive loss to learn such multimodal generative models? It feels somewhat odd to both normalize p(X, Y) and also require unaligned examples as supervision. With an energy based model one can do joint generation, conditional generation (coherence) and synergy studies. A discussion on this would be really helpful!\n\nA minor point, one could also consider the case where “r” is latent, that would actually bring up a very relevant case in vision and language where one crawls the web and the vision and language pairs may be apriori related or unrelated (but it is not known if they are or not). Also, intuitively, why is it that using unrelated data makes us believe that the models should be more data efficient? It would be great to give more intuition on that point. \n\nEqn. 2: The way the equation is written down (given the motivation above) it is not clear that the sum in the second term should be inside the log. Why was this choice made, instead of a more standard contrastive learning approach of summing over the distance with the negative examples (which would make the sum outside the log?). Also, paragraph 1 page 4 is very confusing as the model is talked about as generating when training but as I understand there is no generation when training (only computation of likelihoods?). \n\nAppendix B proof: It appears that the proof might be incorrect as the sum over X that we have in Eqn. 4 is not over all X but over a sample of N examples randomly chosen. It then appears incorrect to claim that this is the same as PMI (as opposed to some kind of an approximation?). \n\nBaseline explanation: It is not clear how the proposed approach is applied to the JMVAE objective (Table. 1), which adds additional terms to the regular ELBO. Are those additional terms also used? Clarification on that would be very helpful. \n\nOverall, it would be very useful to get more insights on why the proposed approach is expected to yield a better multimodal generative model based on the four criteria. Is the idea that generative models without explicitly being given negative supervision are optimistic in which datapoints they consider to be related? How does contrastive training affect \"coverage\" metric as discussed in Vedantam et.al. (Triple ELBO)? Essentially, any insight on when one should and should not do contrastive training would be useful to add to the paper.\n\n**POST REBUTTAL**\nI read through the other reviews and the author rebuttal. I thank the authors for addressing all of my concerns in the rebuttal. Overall, this is an interesting contribution in the multimodal VAE space and would make for a good poster at the conference, and I am happy to keep my original rating for the paper. \n\nIn terms of energy based models here is one recent work which comes to mind which might be useful to extend to multimodal settings: \nDu, Yilun, Shuang Li, and Igor Mordatch. 2020. “Compositional Visual Generation and Inference with Energy Based Models.” arXiv [cs.CV], April. https://arxiv.org/abs/2004.06030.\n(note the quality of generations in Fig. 5)", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an efficient method to train generative models on multimodal data using a contrastive approach. Usually training such models requires significant training data to be able to learn patterns. The authors propose a variational autoencoder approach that enables multimodal learning of models using a data-efficient approach, and shows the effectiveness of the approach on challenging datasets.

The authors have mostly addressed the feedback of the reviewers and done some of the necessary changes to the paper (e.g., adding more results and missing related work). They should make sure to address any lingering concerns about the paper, mentioned by the reviewers in their post-rebuttal feedback.
Decision: Accept (Poster)