Scalable multimodal variational autoencoders with surrogate joint posterior

1 INTRODUCTION . Our world is characterized by many kinds of information . This multimodal information can be used to form common concepts . For instance , by recognizing information , such as the landscape of the beach and the salty scent of the ocean , we can form a concept about the sea . In multimodal learning ( Baltrušaitis et al. , 2018 ) , it has been studied as the learning of joint representations from multimodal data , especially using variational autoencoders ( VAEs ) ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) . The goal of multimodal representation learning with VAEs ( Suzuki et al. , 2016 ; Vedantam et al. , 2018 ; Wu & Goodman , 2018 ; Shi et al. , 2019 ) is to learn all given multimodal data and to infer a joint representation from arbitrary subsets of modalities . This representation is useful as an input to predict the corresponding label information or as an intermediate representation to generate other modalities , i.e. , cross-generations . However , this is difficult because a joint representation must be inferred even when some of the modalities are missing at test time . The most naive way to deal with this difficulty is to have inference models for all possible combinations of modalities , which is exponentially necessary given the number of modalities . A scalable way to address this is to consider the inferences for each modality as experts and combine them to approximate the inferences from an arbitrary subset of modalities . There are two types of methods for aggregating inferences : products of experts ( PoE ) ( Hinton , 2002 ) and mixtures of experts ( MoE ) ; VAEs that use these as approximate posteriors are proposed as MVAE ( Wu & Goodman , 2018 ) and MMVAE ( Shi et al. , 2019 ) , respectively . Each method has different advantages that exist in a trade-off relationship , i.e. , PoE increases the amount of mutual information with joint representation by increasing the number of modalities , while MoE has the effect of encouraging cross-generation between modalities . Sutter et al . ( 2021 ) defined the objective of multimodal VAEs to be that the inference of all possible subsets approximate the true joint posterior , and derived a generalized evidence lower bound ( ELBO ) as a lower bound of that objective . This ELBO can be viewed as using a mixture of products of experts ( MoPoE ) as an aggregation method for inference , which is to generate all modalities from PoE-aggregated inference of arbitrary subsets . In other words , this model , called MoPoE-VAE , tries to achieve the above objective of multimodal VAEs by minimizing all reconstruction and crossgeneration errors in all modality combinations . They showed that this is a generalization that has the advantages of both PoE and MoE . However , these properties of MoPoE-VAE imply that inference may not be sufficiently learned when cross-generation is difficult . Furthermore , the number of paths for this reconstruction and cross-generation in MoPoE-VAE is required to be exponential with respect to the number of modalities . Therefore , we pursue a scalable method that realizes the above objective of multimodal VAEs . In this paper , we propose a different approach to achieve this objective . We first prepare a surrogate joint posterior , which approximates the true joint posterior given all modalities , and then minimize the divergence between the PoE-aggregated inferences and this surrogate posterior . The approximation of surrogate posterior is perform by the usual VAE optimization , and we simultaneously perform this optimization and the divergence minimization above . The advantages of this new approach are twofold . First , if the surrogate model is sufficiently close to the true posterior , then optimizing this objective function explicitly achieves the objective of multimodal VAEs defined in MoPoE , i.e , approximating inferences from arbitrary subsets to the true posterior . Second , unlike MoPoE , this method is not exponentially more expensive . This is because this method is based on divergence minimization of all inference models , rather than reconstruction or cross-generation of all modalities in combination . Although this method requires an additional surrogate model , the size is not exponential as in previous similar methods ( Suzuki et al. , 2016 ; Vedantam et al. , 2018 ) , but only doubles the number of parameters in encoders . Therefore , we call this method a scalable multimodal VAE ( SMVAE ) . Furthermore , to reduce the number of parameters in the surrogate posterior , we propose to use variational gradient origin networks ( VGONs ) ( Bond-Taylor & Willcocks , 2020 ) for training the surrogate posterior . VGONs are deep generative models that can rapidly acquire amortized inference to representations without a trainable network . We call our method that uses VGONs for surrogate posterior SMVGON . Experiments show that the proposed method performs better than conventional methods in inferring joint representations and generating modalities on commonly used multimodal benchmarks . In particular , we show that the proposed method improves the performance in situations where the conventional methods have difficulty in cross-generation . We also confirm that the performance of SMVGONs with VGONs is comparable to that of SMVAEs when the input modalities are relatively simple , without the need for additional inference networks for surrogate posteriors . 2 PRELIMINARY . 2.1 MULTIMODAL VAES . Suppose that we are given an i.i.d . dataset { X ( i ) } Ni=1 , where each example is a set of M modalities X ( i ) = { x ( i ) m } Mm=1 and where each modality x ( i ) m = [ x ( i ) 1m , ... , x ( i ) Dmm ] has its own dimension Dm and domain . We assume that these examples are derived from data distribution pdata ( X ) and that each example X ( i ) has a corresponding common latent concept z ( i ) = [ z ( i ) 1 , ... , z ( i ) j , ... , z ( i ) J ] ∈ R J , i.e. , a joint representation . Given a training set { X ( i ) } Ni=1 , our objective is to infer a joint representation from the subset of modalities1 Xs ⊆ X and to generate another subset X ′ s ⊆ X via that representation . In other words , we aim to obtain an inference p ( z|X ) and a joint distribution p ( X , z ) = ∏ xm∈X p ( xm|z ) p ( z ) using a training set . In this study , we designate the generation when Xs = X ′ s as reconstruction and when Xs ∩X ′ s = ∅ as cross-generation . To achieve this , we use VAEs ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) , which are deep generative models that can learn inference in addition to generative models . Therefore , we can acquire representations of data by inference . The generative model pΘ ( X|z ) = ∏ xm∈X pθm ( xm|z ) , where Θ = { θm } M m=1 is a set of parameters , is parameterized by deep neural networks , and the prior of 1We denote the multimodal set and any subset of it by X and Xs , respectively . We also omit the index of each example for brevity . the latent variable is set as a standard Gaussian p ( z ) = N ( 0 , I ) . The objective function of VAEs is to maximize the marginal log-likelihood of the multimodal dataset : ∑N i=1 log pΘ ( Xi ) . However , since it is tractable to optimize this likelihood directly , we introduce an inference model qΦ ( z|X ) to approximate the posterior pΘ ( z|X ) , and instead optimize the following lower bound of the marginal log-likelihood given X , which is called the ELBO : L ( Θ , Φ ; X ) ≡EqΦ ( z|X ) [ log pΘ ( X|z ) ] −DKL ( qΦ ( z|X ) ||p ( z ) ) . ( 1 ) Here , the optimization of ELBO with respect to the parameters of the inference model corresponds to minimizing the following Kullback-Leiber ( KL ) divergence : DKL ( qΦ ( z|X ) ||pΘ ( z|X ) ) . ( 2 ) In other words , maximizing the ELBO encourages bringing the true posterior closer to the inference model . After optimizing Equation 1 , we aim to infer a joint representation z from an arbitrary subset of modalities Xs . However , this is challenging because Equation 1 only models joint inferences from all modalities . Therefore , for inferences from a subset of modalities Xs , other modality inputs are missing , which results in the inability to infer the representation properly ( Suzuki et al. , 2016 ) . A scalable way to alleviate this difficulty is to consider the inference model for each modality as an expert and approximate the inference model for arbitrary modalities by combining them . There are two ways to combine experts : product and mixture . 2.1.1 AGGREGATED INFERENCE WITH POE AND MOE . A joint inference given all modalities X is expressed as the following using PoE ( Hinton , 2002 ) : qPoE ( z|X ) ∝ p ( z ) ∏ xm∈X qφm ( z|xm ) . ( 3 ) In general , PoE can not be calculated as closed-form , but if we restrict each expert to Gaussian qφm ( z|xm ) = N ( µm , diag ( σ 2 m ) ) ) , then Equation 3 is calculable as qPoE ( z|X ) = N ( µ , diag ( σ2 ) ) , where µj = ∑ m µjmσ −2 jm∑ m σ −2 jm and σ2j = 1∑ m σ −2 jm . Therefore , the Gaussian PoE enables us to infer the representation from subset Xs . However , if we optimize this inference model with Equation 1 , then the inference of arbitrary modalities is not optimized explicitly . Wu & Goodman ( 2018 ) proposed to train ELBOs with PoEaggregated inferences of all modality combinations ∑ Xs∈X L ( Θs , Φs ; Xs ) ; however , since the number of ELBOs increases exponentially , they used the subsampled version . The model that learns PoE inference by this objective function is called MVAE ( Wu & Goodman , 2018 ) . Another method to combine experts is to use MoE : qMoE ( z|X ) = ∑ xm∈X wmqφm ( z|xm ) , ( 4 ) where wm is a mixing coefficient of each expert and where ∑ xm∈X wm = 1 . Shi et al . ( 2019 ) proposed a VAE with MoE-aggregated inference models called MMVAE . There is a trade-off between the approximate posterior with PoE and MoE . PoE can perform inference given any subset and MVAE can learn inference explicitly during training . PoE also has the property that increasing the size of the subset , i.e. , the number of input modalities , leads to an increase in the information content of the inferred latent variables . However , the objective function of MVAE does not guarantee that the inferences from any subset are the same , since it only learns reconstruction and not cross-generation . Sutter et al . ( 2021 ) also pointed out that its objective function by subsampling is not a valid lower bound . MMVAE based on MoE , on the other hand , has a valid ELBO and learns to explicitly encourage cross-generation , which ensures that the inferences from each modality remain consistent . However , this method only learns inferences from a single modality , and increasing the number of modalities does not necessarily increase the information content of the representation .

Summary: The authors propose a new approach to the class of scalable, multimodal VAEs. They propose to use a surrogate joint posterior which, using an additional sum of KL-divergence terms, helps keeping the posterior approximation of subsets of modalities similar to the joint posterior approximation. The proposed method is evaluated on two different datasets, MNIST-SVHN-Text and bimodal CelebA where the proposed method shows good results.