Scalable multimodal variational autoencoders with surrogate joint posterior

1 INTRODUCTION . Our world is characterized by many kinds of information . This multimodal information can be used to form common concepts . For instance , by recognizing information , such as the landscape of the beach and the salty scent of the ocean , we can form a concept about the sea . In multimodal learning ( Baltrušaitis et al. , 2018 ) , it has been studied as the learning of joint representations from multimodal data , especially using variational autoencoders ( VAEs ) ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) . The goal of multimodal representation learning with VAEs ( Suzuki et al. , 2016 ; Vedantam et al. , 2018 ; Wu & Goodman , 2018 ; Shi et al. , 2019 ) is to learn all given multimodal data and to infer a joint representation from arbitrary subsets of modalities . This representation is useful as an input to predict the corresponding label information or as an intermediate representation to generate other modalities , i.e. , cross-generations . However , this is difficult because a joint representation must be inferred even when some of the modalities are missing at test time . The most naive way to deal with this difficulty is to have inference models for all possible combinations of modalities , which is exponentially necessary given the number of modalities . A scalable way to address this is to consider the inferences for each modality as experts and combine them to approximate the inferences from an arbitrary subset of modalities . There are two types of methods for aggregating inferences : products of experts ( PoE ) ( Hinton , 2002 ) and mixtures of experts ( MoE ) ; VAEs that use these as approximate posteriors are proposed as MVAE ( Wu & Goodman , 2018 ) and MMVAE ( Shi et al. , 2019 ) , respectively . Each method has different advantages that exist in a trade-off relationship , i.e. , PoE increases the amount of mutual information with joint representation by increasing the number of modalities , while MoE has the effect of encouraging cross-generation between modalities . Sutter et al . ( 2021 ) defined the objective of multimodal VAEs to be that the inference of all possible subsets approximate the true joint posterior , and derived a generalized evidence lower bound ( ELBO ) as a lower bound of that objective . This ELBO can be viewed as using a mixture of products of experts ( MoPoE ) as an aggregation method for inference , which is to generate all modalities from PoE-aggregated inference of arbitrary subsets . In other words , this model , called MoPoE-VAE , tries to achieve the above objective of multimodal VAEs by minimizing all reconstruction and crossgeneration errors in all modality combinations . They showed that this is a generalization that has the advantages of both PoE and MoE . However , these properties of MoPoE-VAE imply that inference may not be sufficiently learned when cross-generation is difficult . Furthermore , the number of paths for this reconstruction and cross-generation in MoPoE-VAE is required to be exponential with respect to the number of modalities . Therefore , we pursue a scalable method that realizes the above objective of multimodal VAEs . In this paper , we propose a different approach to achieve this objective . We first prepare a surrogate joint posterior , which approximates the true joint posterior given all modalities , and then minimize the divergence between the PoE-aggregated inferences and this surrogate posterior . The approximation of surrogate posterior is perform by the usual VAE optimization , and we simultaneously perform this optimization and the divergence minimization above . The advantages of this new approach are twofold . First , if the surrogate model is sufficiently close to the true posterior , then optimizing this objective function explicitly achieves the objective of multimodal VAEs defined in MoPoE , i.e , approximating inferences from arbitrary subsets to the true posterior . Second , unlike MoPoE , this method is not exponentially more expensive . This is because this method is based on divergence minimization of all inference models , rather than reconstruction or cross-generation of all modalities in combination . Although this method requires an additional surrogate model , the size is not exponential as in previous similar methods ( Suzuki et al. , 2016 ; Vedantam et al. , 2018 ) , but only doubles the number of parameters in encoders . Therefore , we call this method a scalable multimodal VAE ( SMVAE ) . Furthermore , to reduce the number of parameters in the surrogate posterior , we propose to use variational gradient origin networks ( VGONs ) ( Bond-Taylor & Willcocks , 2020 ) for training the surrogate posterior . VGONs are deep generative models that can rapidly acquire amortized inference to representations without a trainable network . We call our method that uses VGONs for surrogate posterior SMVGON . Experiments show that the proposed method performs better than conventional methods in inferring joint representations and generating modalities on commonly used multimodal benchmarks . In particular , we show that the proposed method improves the performance in situations where the conventional methods have difficulty in cross-generation . We also confirm that the performance of SMVGONs with VGONs is comparable to that of SMVAEs when the input modalities are relatively simple , without the need for additional inference networks for surrogate posteriors . 2 PRELIMINARY . 2.1 MULTIMODAL VAES . Suppose that we are given an i.i.d . dataset { X ( i ) } Ni=1 , where each example is a set of M modalities X ( i ) = { x ( i ) m } Mm=1 and where each modality x ( i ) m = [ x ( i ) 1m , ... , x ( i ) Dmm ] has its own dimension Dm and domain . We assume that these examples are derived from data distribution pdata ( X ) and that each example X ( i ) has a corresponding common latent concept z ( i ) = [ z ( i ) 1 , ... , z ( i ) j , ... , z ( i ) J ] ∈ R J , i.e. , a joint representation . Given a training set { X ( i ) } Ni=1 , our objective is to infer a joint representation from the subset of modalities1 Xs ⊆ X and to generate another subset X ′ s ⊆ X via that representation . In other words , we aim to obtain an inference p ( z|X ) and a joint distribution p ( X , z ) = ∏ xm∈X p ( xm|z ) p ( z ) using a training set . In this study , we designate the generation when Xs = X ′ s as reconstruction and when Xs ∩X ′ s = ∅ as cross-generation . To achieve this , we use VAEs ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) , which are deep generative models that can learn inference in addition to generative models . Therefore , we can acquire representations of data by inference . The generative model pΘ ( X|z ) = ∏ xm∈X pθm ( xm|z ) , where Θ = { θm } M m=1 is a set of parameters , is parameterized by deep neural networks , and the prior of 1We denote the multimodal set and any subset of it by X and Xs , respectively . We also omit the index of each example for brevity . the latent variable is set as a standard Gaussian p ( z ) = N ( 0 , I ) . The objective function of VAEs is to maximize the marginal log-likelihood of the multimodal dataset : ∑N i=1 log pΘ ( Xi ) . However , since it is tractable to optimize this likelihood directly , we introduce an inference model qΦ ( z|X ) to approximate the posterior pΘ ( z|X ) , and instead optimize the following lower bound of the marginal log-likelihood given X , which is called the ELBO : L ( Θ , Φ ; X ) ≡EqΦ ( z|X ) [ log pΘ ( X|z ) ] −DKL ( qΦ ( z|X ) ||p ( z ) ) . ( 1 ) Here , the optimization of ELBO with respect to the parameters of the inference model corresponds to minimizing the following Kullback-Leiber ( KL ) divergence : DKL ( qΦ ( z|X ) ||pΘ ( z|X ) ) . ( 2 ) In other words , maximizing the ELBO encourages bringing the true posterior closer to the inference model . After optimizing Equation 1 , we aim to infer a joint representation z from an arbitrary subset of modalities Xs . However , this is challenging because Equation 1 only models joint inferences from all modalities . Therefore , for inferences from a subset of modalities Xs , other modality inputs are missing , which results in the inability to infer the representation properly ( Suzuki et al. , 2016 ) . A scalable way to alleviate this difficulty is to consider the inference model for each modality as an expert and approximate the inference model for arbitrary modalities by combining them . There are two ways to combine experts : product and mixture . 2.1.1 AGGREGATED INFERENCE WITH POE AND MOE . A joint inference given all modalities X is expressed as the following using PoE ( Hinton , 2002 ) : qPoE ( z|X ) ∝ p ( z ) ∏ xm∈X qφm ( z|xm ) . ( 3 ) In general , PoE can not be calculated as closed-form , but if we restrict each expert to Gaussian qφm ( z|xm ) = N ( µm , diag ( σ 2 m ) ) ) , then Equation 3 is calculable as qPoE ( z|X ) = N ( µ , diag ( σ2 ) ) , where µj = ∑ m µjmσ −2 jm∑ m σ −2 jm and σ2j = 1∑ m σ −2 jm . Therefore , the Gaussian PoE enables us to infer the representation from subset Xs . However , if we optimize this inference model with Equation 1 , then the inference of arbitrary modalities is not optimized explicitly . Wu & Goodman ( 2018 ) proposed to train ELBOs with PoEaggregated inferences of all modality combinations ∑ Xs∈X L ( Θs , Φs ; Xs ) ; however , since the number of ELBOs increases exponentially , they used the subsampled version . The model that learns PoE inference by this objective function is called MVAE ( Wu & Goodman , 2018 ) . Another method to combine experts is to use MoE : qMoE ( z|X ) = ∑ xm∈X wmqφm ( z|xm ) , ( 4 ) where wm is a mixing coefficient of each expert and where ∑ xm∈X wm = 1 . Shi et al . ( 2019 ) proposed a VAE with MoE-aggregated inference models called MMVAE . There is a trade-off between the approximate posterior with PoE and MoE . PoE can perform inference given any subset and MVAE can learn inference explicitly during training . PoE also has the property that increasing the size of the subset , i.e. , the number of input modalities , leads to an increase in the information content of the inferred latent variables . However , the objective function of MVAE does not guarantee that the inferences from any subset are the same , since it only learns reconstruction and not cross-generation . Sutter et al . ( 2021 ) also pointed out that its objective function by subsampling is not a valid lower bound . MMVAE based on MoE , on the other hand , has a valid ELBO and learns to explicitly encourage cross-generation , which ensures that the inferences from each modality remain consistent . However , this method only learns inferences from a single modality , and increasing the number of modalities does not necessarily increase the information content of the representation .

Summary: SMVAE aims to solve the challenge of combining multiple modalities into a coherent latent space using VAE framework. SMVAE is based on MoPoE-VAE model, the main architectural change is replacing the MoE rule with minimising the KL-divergence between the latent space distribution generated by the PoE combined experts and the joint encoder.    (The reviewer is sorry for the slight delay for submission as he is still on vacation and marked november third as deadline for submission. In the interest of time,the following is just a short summary of my notes)