Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Models

1 INTRODUCTION To comprehensively describe concepts in the real world , humans collect multiple perceptual signals of the same object such as image , sound , text and video . We refer to each of these media as a modality , and a collection of different media featuring the same underlying concept is characterised as multimodal data . Learning from multiple modalities has been shown to yield more generalisable representations ( Zhang et al. , 2020 ; Guo et al. , 2019 ; Yildirim , 2014 ) , as different modalities are often complimentary in content while overlapping for common abstract concept . Despite the motivation , it is worth noting that the multimodal framework is not exactly data-efficient—constructing a suitable dataset requires a lot of “ annotated ” unimodal data , as we need to ensure that each multimodal pair is related in a meaningful way . The situation is worse when we consider more complicated multimodal settings such as language–vision , where one-to-one or one-to-many correspondence between instances of the two datasets are required , due to the difficulty in categorising data such that commonality amongst samples is preserved within categories . See Figure 1 for an example from the CUB dataset ( Welinder et al. , a ) ; although the same species of bird is featured in both image-caption pairs , their content differs considerably . It would be unreasonable to apply the caption from one to describe the bird depicted in the other , necessitating one-to-one correspondence between images and captions . However , the scope of multimodal learning has been limited to leveraging the commonality between “ related ” pairs , while largely ignoring “ unrelated ” samples potentially available in any multimodal dataset—constructed through random pairing between modalities ( Figure 3 ) . We posit that if a distinction can be established between the “ related ” and “ unrelated ” observations within a multimodal dataset , we could greatly reduce the amount of related data required for effective learning . Figure 2 formalises this proposal . Multimodal generative models in previous work ( Figure 2a ) typically assumes one latent variable z that always generates related multimodal pair ( x , y ) . In this work ( Figure 2b ) , we introduce an additional Bernoulli random variable r that dictates the “ relatedness ” between x and y through z , where x and y are related when r = 1 , and unrelated when r = 0 . ∗work done while at Oxford ( a ) Previous ( b ) Ours Figure 2 : Graphical models for multimodal generative process . While r can encode different dependencies , here we make the simplifying assumption that the Pointwise Mutual Information ( PMI ) between x and y should be high when r = 1 , and low when r = 0 . Intuitively , this can be achieved by adopting a max-margin metric . We therefore propose to train the generative moels with a novel contrastive-style loss ( Hadsell et al. , 2006 ; Weinberger et al. , 2005 ) , and demonstrate the effectiveness of our proposed method from a few different perspectives : Improved multimodal learning : showing improved multimodal learning for various state-of-the-art multimodal generative models on two challenging multimodal datasets . This is evaluated on four different metrics ( Shi et al. , 2019 ) summarised in § 4.2 ; Data efficiency : learning generative models under the contrastive framework requires only 20 % of the data needed in baseline methods to achieve similar performance—holding true across different models , datasets and metrics ; Label propagation : the contrastive loss encourages a larger discrepency between related and unrelated data , making it possible to directly identify related samples using the PMI between observations . We show that these data pairs can be used to further improve the learning of the generative model . 2 RELATED WORK . Contrastive loss Our work aims to encourage data-efficient multimodal generative-model learning using a popular representation learning metric—contrastive loss ( Hadsell et al. , 2006 ; Weinberger et al. , 2005 ) . There has been many successful applications of contrastive loss to a range of different tasks , such as contrastive predictive coding for time series data ( van den Oord et al. , 2018 ) , image classification ( Hénaff , 2020 ) , noise contrastive estimation for vector embeddings of words ( Mnih and Kavukcuoglu , 2013 ) , as well as a range of frameworks such as DIM ( Hjelm et al. , 2019 ) , MoCo ( He et al. , 2020 ) , SimCLR ( Chen et al. , 2020 ) for more general visual-representation learning . The features learned by contrastive loss also perform well when applied to different downstream tasks — their ability to generalise is further analysed in ( Wang and Isola , 2020 ) using quantifiable metrics for alignment and uniformity . Contrastive methods have also been employed under a generative-model setting , but typically on generative adversarial networks ( GANs ) to either preserve or identify factors-of-variations in their inputs . For instance , SiGAN ( Hsu et al. , 2019 ) uses a contrastive loss to preserve identity for faceimage hallucination from low-resolution photos , while ( Yildirim et al. , 2018 ) uses a contrastive loss to disentangle the factors of variations in the latent code of GANs . We here employ a contrastive loss in a distinct setting of multimodal generative model learning , that , as we will show with our experiments and analyses promotes better , more robust representation learning . Multimodal VAEs We also demonstrate that our approach is applicable across different approaches to learning multimodal generative models . To do so , we first summarise past work on multimodal VAE into two categories based on the modelling choice of approximated posterior qΦ ( z|x , y ) : a ) Explicit joint models : qΦ as single joint encoder qΦ ( z|x , y ) . Example work in this area include JMVAE ( Suzuki et al . ) , triple ELBO ( Vedantam et al. , 2018 ) and MFM ( Tsai et al. , 2019 ) . Since the joint encoder require multimodal pair ( x , y ) as input , these approaches typically require additional modelling components and/or inference steps to deal with missing modality at test time ; in fact , all three approaches propose to train unimodal VAEs on top of the joint model that handles data from each modality independently . b ) Factorised joint models : qΦ as factored encoders qΦ ( z|x , y ) = f ( qφx ( z|x ) , qφy ( z|y ) ) . This was first seen in Wu and Goodman ( 2018 ) , as the MVAE model with f defined as a product of experts ( POE ) , i.e . qΦ ( z|x , y ) = qφx ( z|x ) qφy ( z|y ) p ( z ) , allowing for cross-modality generation without extra modelling components . Particularly , the MVAE caters to settings where data was not guaranteed to be always related , and where additional modalities were , in terms of information content , subsets of a primary data source—such as images and their class labels . Alternately , Shi et al . ( 2019 ) explored an approach that explicitly leveraged the availability of related/paired data , motivated by arguments from embodied cognition of the world . They propose the MMVAE model , which additionally differs from the MVAE model in its choice of posterior approximation—where f is modelled as the mixture of experts ( MOE ) of unimodal posteriors—to ameliorate shortcomings to do with precision miscalibration of the POE . Furthermore , Shi et al . ( 2019 ) also posit four criteria that a multimodal VAE should satisfy , which we adopt in this work to evaluate the performance of our models . Weakly-supervised learning Using generative models for label propagation ( see § 4.5 ) is a form of weak supervision . Commonly seen approaches for weakly-supervised training with incomplete data include ( 1 ) graph-based methods ( such as minimum cut ) ( Zhou et al. , 2003 ; Zhu et al. , 2003 ; Blum and Chawla , 2001 ) , ( 2 ) low-density separation methods ( Joachims , 1999 ; Burkhart and Shan , 2020 ) and ( 3 ) disagreement-based models ( Blum and Mitchell , 1998 ; Zhou and Li , 2005 ; 2010 ) . However , ( 1 ) and ( 2 ) suffers from scalability issues due to computational inefficiency and optimisation complexity , while ( 3 ) works well for many different tasks but requires training an ensemble of learners . The use of generative models for weakly supervised learning has also been explored in Nigam et al . ( 2000 ) ; Miller and Uyar ( 1996 ) , where labels are estimated using expectation-maximisation ( EM ) ( Dempster et al. , 1977 ) for instances that are unlabelled . Notably , models trained with our contrastive objective does not need EM to leverage unlabelled data ( see § 4.5 ) to determine the relatedness of two examples we only need to compute a threshold ( estimation of PMI ) using the trained model . 3 METHODOLOGY . Given data over observations from two modalties ( x , y ) , one can learn a multimodal VAE targetting pΘ ( x , y , z ) = p ( z ) pθx ( x|z ) pθy ( y|z ) , where pθ ( ·|z ) are deep neural networks ( decoders ) parametrised by Θ = { θx , θy } . To maximise the joint marginal likelihood log pΘ ( x , y ) , one approximates the intractable model posterior pΘ ( z|x , y ) with a variational posterior qΦ ( z|x , y ) , allowing us to optimise a variational evidence lower bound ( ELBO ) , defined as log pΘ ( x , y ) ≥ Ez∼ qΦ ( z|x , y ) [ log pΘ ( z , x , y ) qΦ ( z | x , y ) ] = ELBO ( x , y ) . ( 1 ) This leaves open the question of how to model the approximatd posterior qΦ ( z|x , y ) . As mentioned in § 2 , there are two schools of thinking , namely explicit joint model such as JMVAE ( Suzuki et al . ) and factorised joint model including MVAE ( Wu and Goodman , 2018 ) and MMVAE ( Shi et al. , 2019 ) . In this work we demonstrate the effectiveness of our approach across all these models . 3.1 CONTRASTIVE LOSS FOR “ RELATEDNESS ” LEARNING . Where prior work always assumes ( x , y ) to be related , we introduce a relatedness variable r explicitly capturing this aspect . Our approach is motivated by the characteristics of pointwise mutual information ( PMI ) between related and unrelated observations across modalities : Hypothesis 3.1 . Let ( x , y ) ∼ pΘ ( x , y ) be a related data pair from two modalities , and let y′ denote a data point unrelated to x . Then , the pointwise mutual information I ( x , y ) > I ( x , y′ ) . ( a ) Related ( b ) Unrelated in PMI between related and unrelated pairs , i.e . I ( x , y ) − I ( x , y′ ) . We show in Appendix A that this is equivalent to maximising the difference between the joint marginal likelihoods of related and unrelated pairs ( Figure 3a and 3b ) . This casts multimodal learning as max-margin optimisation , with the contrastive ( triplet ) loss as a natural choice of objective : LC ( x , y , y′ ) = d ( x , y ) −d ( x , y′ ) +m . Intuitively , LC attempts to make distance d between a positive pair ( x , y ) smaller than the distance between a negative pair ( x , y′ ) by margin m. We adopt this loss to our objective by omitting margin m and replacing d with the ( negative ) joint marginal likelihood − log pΘ . Following Song et al . ( 2016 ) , with N negative samples { y′i } Ni=1 , we have LC ( x , Y ) = − log pΘ ( x , y ) + log N∑ i=1 pΘ ( x , y ′ i ) . ( 2 ) We choose to put the sum overN within the log following conventions in previous work on contrastive loss ( van den Oord et al. , 2018 ; Hjelm et al. , 2019 ; Chen et al. , 2020 ) . As the loss is asymmetric , one can average over LC ( y , X ) and LC ( x , Y ) to ensure negative samples in both modalities are accounted for—giving our contrastive objective : LC ( x , y ) = 1 2 { LC ( x , Y ) +LC ( y , X ) } =−log pΘ ( x , y ) ︸ ︷︷ ︸ 1 + 1 2 log N∑ x′i=1 pΘ ( x ′ , y ) + log N∑ y′i=1 pΘ ( x , y ′ ) ︸ ︷︷ ︸ 2 ( 3 ) Note that since only the joint marginal likelihood terms are needed in ( 3 ) , the contrastive learning framework can be directly applied to any multimodal generative model without needing extra components . We also show our contrastive framework applies when the number of modalities is more than two ( cf . Appendix C ) . Figure 4 : log p ( x , y ) of imitation , unrelated digits and random noise , where p = N ( mx , my ) . Dissecting LC Although similar to the VAE ( 1 ) , our objective ( 3 ) directly maximises pΘ ( x , y ) in 1 . LC by itself is not a effective objective as 2 in LC minimises pΘ ( x , y ) , which can overpower the effect of 1 during training . We intuit this phenomenon using a simple example in Figure 4 with natural images , showing the log likelihood of log p ( x , y ) in column 2 , 3 , 4 ( green ) on a Gaussian distribution N ( ( mx , my ) ; c ) , with the images in the first column of Figure 4 as means and with constant variance c. While achieving high log p ( x , y ) requires matching ( mx , my ) ( col 2 ) , we see both unrelated digits ( col 3 ) and noise ( col 4 ) can lead to ( comparatively ) poor joint log likelihoods . This indicates that the generative model need not generate valid , unrelated images to minimise 2 —generating noise would have roughly the same effect on log likelihood . As a result , the model can trivially minimise LC by generating noise that minimises 2 instead of accurate reconstruction that maximises 1 . This learning dynamic is verified empirically , as we show in Figure 9 in Appendix D : optimising LC by itself results in the loss approaching 0 within the first 100 iterations , and both 1 and 2 takes on extremely low values , resulting in a model that generates random noise . Final objective To mitigate this issue , we need to ensure that minimising 2 does not overpower maximising 1 . We hence introduce a hyperparameter γ on 1 to upweight the maximisation of the marginal likelihood , with the final objective to minimise LC ( x , y ) = −γ log pΘ ( x , y ) ︸ ︷︷ ︸ 1 + 1 2 log N∑ x′i=1 pΘ ( x ′ , y ) + log N∑ y′i=1 pΘ ( x , y ′ ) ︸ ︷︷ ︸ 2 , γ > 1 . ( 4 ) We conduct ablation studies on the effect of γ in Appendix E , noting that larger γ encourages better quality of generation and more stable training in some cases , while models trained with smaller γ are better at predicting “ relatedness ” between multimodal samples . We also note that optimising ( 4 ) maximises the pointwise mutual information I ( x , y ) ; see Appendix B for a proof .

Summary: This work presents a generative model for multimodal learning. The paper maximizes or minimizes the pointwise mutual information between data from two modalities considering a novel random variable relatedness to dictates if data are related or not.  This is realized by casting multimodal learning as max-margin optimization with the contrastive loss for the objective. For the optimization, the paper considers the IWAE estimator. As per the experiments, the paper considers MNIST-SVHN and CUB Image-Captions dataset and perform evaluations across four metrics. Using the experiments, the paper demonstrates that the proposed approach improves multimodal learning, data-efficient learning, and label propagation.