Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Models

1 INTRODUCTION To comprehensively describe concepts in the real world , humans collect multiple perceptual signals of the same object such as image , sound , text and video . We refer to each of these media as a modality , and a collection of different media featuring the same underlying concept is characterised as multimodal data . Learning from multiple modalities has been shown to yield more generalisable representations ( Zhang et al. , 2020 ; Guo et al. , 2019 ; Yildirim , 2014 ) , as different modalities are often complimentary in content while overlapping for common abstract concept . Despite the motivation , it is worth noting that the multimodal framework is not exactly data-efficient—constructing a suitable dataset requires a lot of “ annotated ” unimodal data , as we need to ensure that each multimodal pair is related in a meaningful way . The situation is worse when we consider more complicated multimodal settings such as language–vision , where one-to-one or one-to-many correspondence between instances of the two datasets are required , due to the difficulty in categorising data such that commonality amongst samples is preserved within categories . See Figure 1 for an example from the CUB dataset ( Welinder et al. , a ) ; although the same species of bird is featured in both image-caption pairs , their content differs considerably . It would be unreasonable to apply the caption from one to describe the bird depicted in the other , necessitating one-to-one correspondence between images and captions . However , the scope of multimodal learning has been limited to leveraging the commonality between “ related ” pairs , while largely ignoring “ unrelated ” samples potentially available in any multimodal dataset—constructed through random pairing between modalities ( Figure 3 ) . We posit that if a distinction can be established between the “ related ” and “ unrelated ” observations within a multimodal dataset , we could greatly reduce the amount of related data required for effective learning . Figure 2 formalises this proposal . Multimodal generative models in previous work ( Figure 2a ) typically assumes one latent variable z that always generates related multimodal pair ( x , y ) . In this work ( Figure 2b ) , we introduce an additional Bernoulli random variable r that dictates the “ relatedness ” between x and y through z , where x and y are related when r = 1 , and unrelated when r = 0 . ∗work done while at Oxford ( a ) Previous ( b ) Ours Figure 2 : Graphical models for multimodal generative process . While r can encode different dependencies , here we make the simplifying assumption that the Pointwise Mutual Information ( PMI ) between x and y should be high when r = 1 , and low when r = 0 . Intuitively , this can be achieved by adopting a max-margin metric . We therefore propose to train the generative moels with a novel contrastive-style loss ( Hadsell et al. , 2006 ; Weinberger et al. , 2005 ) , and demonstrate the effectiveness of our proposed method from a few different perspectives : Improved multimodal learning : showing improved multimodal learning for various state-of-the-art multimodal generative models on two challenging multimodal datasets . This is evaluated on four different metrics ( Shi et al. , 2019 ) summarised in § 4.2 ; Data efficiency : learning generative models under the contrastive framework requires only 20 % of the data needed in baseline methods to achieve similar performance—holding true across different models , datasets and metrics ; Label propagation : the contrastive loss encourages a larger discrepency between related and unrelated data , making it possible to directly identify related samples using the PMI between observations . We show that these data pairs can be used to further improve the learning of the generative model . 2 RELATED WORK . Contrastive loss Our work aims to encourage data-efficient multimodal generative-model learning using a popular representation learning metric—contrastive loss ( Hadsell et al. , 2006 ; Weinberger et al. , 2005 ) . There has been many successful applications of contrastive loss to a range of different tasks , such as contrastive predictive coding for time series data ( van den Oord et al. , 2018 ) , image classification ( Hénaff , 2020 ) , noise contrastive estimation for vector embeddings of words ( Mnih and Kavukcuoglu , 2013 ) , as well as a range of frameworks such as DIM ( Hjelm et al. , 2019 ) , MoCo ( He et al. , 2020 ) , SimCLR ( Chen et al. , 2020 ) for more general visual-representation learning . The features learned by contrastive loss also perform well when applied to different downstream tasks — their ability to generalise is further analysed in ( Wang and Isola , 2020 ) using quantifiable metrics for alignment and uniformity . Contrastive methods have also been employed under a generative-model setting , but typically on generative adversarial networks ( GANs ) to either preserve or identify factors-of-variations in their inputs . For instance , SiGAN ( Hsu et al. , 2019 ) uses a contrastive loss to preserve identity for faceimage hallucination from low-resolution photos , while ( Yildirim et al. , 2018 ) uses a contrastive loss to disentangle the factors of variations in the latent code of GANs . We here employ a contrastive loss in a distinct setting of multimodal generative model learning , that , as we will show with our experiments and analyses promotes better , more robust representation learning . Multimodal VAEs We also demonstrate that our approach is applicable across different approaches to learning multimodal generative models . To do so , we first summarise past work on multimodal VAE into two categories based on the modelling choice of approximated posterior qΦ ( z|x , y ) : a ) Explicit joint models : qΦ as single joint encoder qΦ ( z|x , y ) . Example work in this area include JMVAE ( Suzuki et al . ) , triple ELBO ( Vedantam et al. , 2018 ) and MFM ( Tsai et al. , 2019 ) . Since the joint encoder require multimodal pair ( x , y ) as input , these approaches typically require additional modelling components and/or inference steps to deal with missing modality at test time ; in fact , all three approaches propose to train unimodal VAEs on top of the joint model that handles data from each modality independently . b ) Factorised joint models : qΦ as factored encoders qΦ ( z|x , y ) = f ( qφx ( z|x ) , qφy ( z|y ) ) . This was first seen in Wu and Goodman ( 2018 ) , as the MVAE model with f defined as a product of experts ( POE ) , i.e . qΦ ( z|x , y ) = qφx ( z|x ) qφy ( z|y ) p ( z ) , allowing for cross-modality generation without extra modelling components . Particularly , the MVAE caters to settings where data was not guaranteed to be always related , and where additional modalities were , in terms of information content , subsets of a primary data source—such as images and their class labels . Alternately , Shi et al . ( 2019 ) explored an approach that explicitly leveraged the availability of related/paired data , motivated by arguments from embodied cognition of the world . They propose the MMVAE model , which additionally differs from the MVAE model in its choice of posterior approximation—where f is modelled as the mixture of experts ( MOE ) of unimodal posteriors—to ameliorate shortcomings to do with precision miscalibration of the POE . Furthermore , Shi et al . ( 2019 ) also posit four criteria that a multimodal VAE should satisfy , which we adopt in this work to evaluate the performance of our models . Weakly-supervised learning Using generative models for label propagation ( see § 4.5 ) is a form of weak supervision . Commonly seen approaches for weakly-supervised training with incomplete data include ( 1 ) graph-based methods ( such as minimum cut ) ( Zhou et al. , 2003 ; Zhu et al. , 2003 ; Blum and Chawla , 2001 ) , ( 2 ) low-density separation methods ( Joachims , 1999 ; Burkhart and Shan , 2020 ) and ( 3 ) disagreement-based models ( Blum and Mitchell , 1998 ; Zhou and Li , 2005 ; 2010 ) . However , ( 1 ) and ( 2 ) suffers from scalability issues due to computational inefficiency and optimisation complexity , while ( 3 ) works well for many different tasks but requires training an ensemble of learners . The use of generative models for weakly supervised learning has also been explored in Nigam et al . ( 2000 ) ; Miller and Uyar ( 1996 ) , where labels are estimated using expectation-maximisation ( EM ) ( Dempster et al. , 1977 ) for instances that are unlabelled . Notably , models trained with our contrastive objective does not need EM to leverage unlabelled data ( see § 4.5 ) to determine the relatedness of two examples we only need to compute a threshold ( estimation of PMI ) using the trained model . 3 METHODOLOGY . Given data over observations from two modalties ( x , y ) , one can learn a multimodal VAE targetting pΘ ( x , y , z ) = p ( z ) pθx ( x|z ) pθy ( y|z ) , where pθ ( ·|z ) are deep neural networks ( decoders ) parametrised by Θ = { θx , θy } . To maximise the joint marginal likelihood log pΘ ( x , y ) , one approximates the intractable model posterior pΘ ( z|x , y ) with a variational posterior qΦ ( z|x , y ) , allowing us to optimise a variational evidence lower bound ( ELBO ) , defined as log pΘ ( x , y ) ≥ Ez∼ qΦ ( z|x , y ) [ log pΘ ( z , x , y ) qΦ ( z | x , y ) ] = ELBO ( x , y ) . ( 1 ) This leaves open the question of how to model the approximatd posterior qΦ ( z|x , y ) . As mentioned in § 2 , there are two schools of thinking , namely explicit joint model such as JMVAE ( Suzuki et al . ) and factorised joint model including MVAE ( Wu and Goodman , 2018 ) and MMVAE ( Shi et al. , 2019 ) . In this work we demonstrate the effectiveness of our approach across all these models . 3.1 CONTRASTIVE LOSS FOR “ RELATEDNESS ” LEARNING . Where prior work always assumes ( x , y ) to be related , we introduce a relatedness variable r explicitly capturing this aspect . Our approach is motivated by the characteristics of pointwise mutual information ( PMI ) between related and unrelated observations across modalities : Hypothesis 3.1 . Let ( x , y ) ∼ pΘ ( x , y ) be a related data pair from two modalities , and let y′ denote a data point unrelated to x . Then , the pointwise mutual information I ( x , y ) > I ( x , y′ ) . ( a ) Related ( b ) Unrelated in PMI between related and unrelated pairs , i.e . I ( x , y ) − I ( x , y′ ) . We show in Appendix A that this is equivalent to maximising the difference between the joint marginal likelihoods of related and unrelated pairs ( Figure 3a and 3b ) . This casts multimodal learning as max-margin optimisation , with the contrastive ( triplet ) loss as a natural choice of objective : LC ( x , y , y′ ) = d ( x , y ) −d ( x , y′ ) +m . Intuitively , LC attempts to make distance d between a positive pair ( x , y ) smaller than the distance between a negative pair ( x , y′ ) by margin m. We adopt this loss to our objective by omitting margin m and replacing d with the ( negative ) joint marginal likelihood − log pΘ . Following Song et al . ( 2016 ) , with N negative samples { y′i } Ni=1 , we have LC ( x , Y ) = − log pΘ ( x , y ) + log N∑ i=1 pΘ ( x , y ′ i ) . ( 2 ) We choose to put the sum overN within the log following conventions in previous work on contrastive loss ( van den Oord et al. , 2018 ; Hjelm et al. , 2019 ; Chen et al. , 2020 ) . As the loss is asymmetric , one can average over LC ( y , X ) and LC ( x , Y ) to ensure negative samples in both modalities are accounted for—giving our contrastive objective : LC ( x , y ) = 1 2 { LC ( x , Y ) +LC ( y , X ) } =−log pΘ ( x , y ) ︸ ︷︷ ︸ 1 + 1 2 log N∑ x′i=1 pΘ ( x ′ , y ) + log N∑ y′i=1 pΘ ( x , y ′ ) ︸ ︷︷ ︸ 2 ( 3 ) Note that since only the joint marginal likelihood terms are needed in ( 3 ) , the contrastive learning framework can be directly applied to any multimodal generative model without needing extra components . We also show our contrastive framework applies when the number of modalities is more than two ( cf . Appendix C ) . Figure 4 : log p ( x , y ) of imitation , unrelated digits and random noise , where p = N ( mx , my ) . Dissecting LC Although similar to the VAE ( 1 ) , our objective ( 3 ) directly maximises pΘ ( x , y ) in 1 . LC by itself is not a effective objective as 2 in LC minimises pΘ ( x , y ) , which can overpower the effect of 1 during training . We intuit this phenomenon using a simple example in Figure 4 with natural images , showing the log likelihood of log p ( x , y ) in column 2 , 3 , 4 ( green ) on a Gaussian distribution N ( ( mx , my ) ; c ) , with the images in the first column of Figure 4 as means and with constant variance c. While achieving high log p ( x , y ) requires matching ( mx , my ) ( col 2 ) , we see both unrelated digits ( col 3 ) and noise ( col 4 ) can lead to ( comparatively ) poor joint log likelihoods . This indicates that the generative model need not generate valid , unrelated images to minimise 2 —generating noise would have roughly the same effect on log likelihood . As a result , the model can trivially minimise LC by generating noise that minimises 2 instead of accurate reconstruction that maximises 1 . This learning dynamic is verified empirically , as we show in Figure 9 in Appendix D : optimising LC by itself results in the loss approaching 0 within the first 100 iterations , and both 1 and 2 takes on extremely low values , resulting in a model that generates random noise . Final objective To mitigate this issue , we need to ensure that minimising 2 does not overpower maximising 1 . We hence introduce a hyperparameter γ on 1 to upweight the maximisation of the marginal likelihood , with the final objective to minimise LC ( x , y ) = −γ log pΘ ( x , y ) ︸ ︷︷ ︸ 1 + 1 2 log N∑ x′i=1 pΘ ( x ′ , y ) + log N∑ y′i=1 pΘ ( x , y ′ ) ︸ ︷︷ ︸ 2 , γ > 1 . ( 4 ) We conduct ablation studies on the effect of γ in Appendix E , noting that larger γ encourages better quality of generation and more stable training in some cases , while models trained with smaller γ are better at predicting “ relatedness ” between multimodal samples . We also note that optimising ( 4 ) maximises the pointwise mutual information I ( x , y ) ; see Appendix B for a proof .

Summary:  The paper proposes a contrastive objective that (1) minimizes the distance between "related" samples while (2) maximizing the distance between randomly paired samples. Existing multimodal VAEs optimize (1) via different multimodal ELBOs. The novelty lies in the optimization of (2) which can further benefit from unimodal samples for which no "related" samples of the other modality are available---this can be viewed as a semi-supervised approach for weakly-supervised multimodal data.  For the estimation of (2), the paper experiments with two different estimators, IWAE and CUBO.