Deep Generative Models for learning Coherent Latent Representations from Multi-Modal Data

1 INTRODUCTION . Auto Encoder ( AE ) , Variational Auto Encoder ( VAE ) , and more recently Disentangled Variational Auto Encoder ( β-VAE ) have a considerable impact on the field of data-driven leaning of generative models . Furthermore , recent investigations have shown the fruitful applicability to deep reinforcement learning ( DRL ) as well as bi-directionally exchange of multi-modal data . VAEs tend to encode the data into latent space features that are ( ideally ) linearly separable as shown by Higgins et al . ( 2017a ) . They also allow the discovery of generative joint models ( e.g . Suzuki et al . ( 2017 ) ) , as well as zero-shot domain transfer in DRL as shown by Higgins et al . ( 2017b ) . However , a good generative model should not just generate good data and achieve a good quantitative score , but also gives a coherent and expressive latent space representation . This property is decisive for multi-modal approaches if the data shows correlation , as it is the case for every sensor setup designed for sensor fusion . With this contribution , we investigate the characteristic of the latent space as well as the quantitative features for existing multi-modal VAEs . Furthermore , we propose a novel approach to build and train a novel multi-modal VAE ( M2VAE ) which comprises the complete marginal joint log-likelihood without simplifying assumptions . As our objective is the consideration of raw multi-modal sensor data , we also propose an approach to generate correlated multi-modal datasets from available uni-modal ones . Lastly , we draw connections to in-place sensor fusion and epistemic ( ambiguity-resolving ) active-sensing . Section 2 comprises the related work on multi-modal VAEs . Our comprehensive approach ( i.e . M2VAE ) is given in Sec . 3 . Furthermore , we describe multi-modal datasets as well as the generation of correlated sets in Sec . 4 which are evaluated in Sec . 5 . Finally , we conclude our work in Sec . 6 . 2 RELATED WORK . Variational auto encoder ( VAE ) combine neural networks with variational inference to allow unsupervised learning of complicated distributions according to the graphical model shown in Figure 1 ( left ) . A Da-dimensional observation a is modeled in terms of a Dz-dimensional latent vector z using a probabilistic decoder pθa ( z ) with parameters θ . To generate the corresponding embedding z from observation a , a probabilistic encoder network with qφa ( z ) is being provided which parametrizes the posterior distribution from which z is sampled . The encoder and decoder , given by neural networks , are trained jointly to bring a close to an a′ under the constraint that an approximate distribution needs to be close to a prior p ( z ) and hence inference is basically learned during training . The specific objective of VAEs is the maximization of the marginal distribution p ( a ) =∫ pθ ( a|z ) p ( z ) da . Because this distribution is intractable , the model is instead trained via stochastic gradient variational Bayes ( SGVB ) by maximizing the evidence lower bound ( ELBO ) L of the marginal log-likelihood log p ( a ) : = La as La≥L=−DKL ( qφ ( z|a ) ‖p ( z ) ) ︸ ︷︷ ︸ Regularization +Eqφ ( z|a ) log ( pθ ( a|z ) ) ︸ ︷︷ ︸ Reconstruction . ( 1 ) This approach proposed by Kingma & Welling ( 2013 ) is used in settings where only a single modality a is present in order to find a latent encoding z ( c.f . Figure 1 ( left ) ) . In the following chapters , we give a briefly comprise related work by means of multi-modal VAEs . Further , we stress the concept of two joint multi-modal approaches to derive the later proposed variational Auto Encoder ( VAE ) . 2.1 MULTI-MODAL AUTO ENCODER . Given a set of modalities M = { a , b , c , . . . } , multi-modal variants of Variational Auto Encoders ( VAE ) have been applied to train generative models for multi-directional reconstruction ( i.e . generation of missing data ) or feature extraction . Variants are conditional VAEs ( CVAE ) and conditional multi-modal autoencoders ( CMMA ) , with the lack in bi-directional reconstruction ( Sohn et al . ( 2015 ) ; Pandey & Dukkipati ( 2017 ) ) . BiVCCA by Wang et al . ( 2016 ) trains two VAEs together with interacting inference networks to facilitate two-way reconstruction with the lack of directly modeling the joint distribution . Models , that are derived from the variation of information ( VI ) with the objective to estimate the joint distribution with the capabilities of multi-directional reconstruction were recently introduced by Suzuki et al . ( 2017 ) . Vedantam et al . ( 2017 ) introduce another objective for the bi-modal VAE , which they call the triplet ELBO ( tVAE ) . Furthermore , multi-modal stacked Auto Encoders ( AE ) are a variant of combining the latent spaces of various AEs ( Larochelle et al . ( 2007 ) ; Ranzato et al . ( 2006 ) ) which can also be applied to the reconstruction of missing modalities ( Ngiam et al . ( 2011 ) ; Cadena et al . ( 2016 ) ) . However , while Suzuki et al . ( 2017 ) and Vedantam et al . ( 2017 ) argue that training of the full multi-modal VAE is intractable , because of the 2|M|−1 modality subsets of inference networks , we show that training the full joint model estimates the most expressive latent embeddings . 2.1.1 JOINT MULTI-MODAL VARIATIONAL AUTO ENCODER . When more than one modality is available , e.g . a and b as shown in Figure 1 ( mid . ) , the derivation of the ELBO LJ for a marginal joint log-likelihood log p ( a ) : = LJ is straight forward : LJ ≥ LJ = −DKL ( qφab ( z|a , b ) ‖p ( z ) ) ︸ ︷︷ ︸ Regularization +Eqφab ( z|a , b ) log ( pθa ( a|z ) ) ︸ ︷︷ ︸ Reconstruction wrt . a +Eqφab ( z|a , b ) log ( pθb ( b|z ) ) ︸ ︷︷ ︸ Reconstruction wrt . b ( 2 ) However , it is not clear how to perform inference if the dataset consists of samples lacking from modalities ( e.g . for samples i and k : ( ai , ∅ ) and ( ∅ , bk ) ) . Ngiam et al . ( 2011 ) propose training of a bimodal deep auto encoder using an augmented dataset with additional examples that have only a single-modality as input . We , therefore , name the resulting model of Eq . 2 joint multi-modal VAE-Zero ( JMVAE-Zero ) . 2.1.2 JOINT MULTI-MODAL VARIATIONAL AUTO ENCODER FROM VARIATION OF INFORMATION . While the former approach can not directly be applied to missing modalities , Suzuki et al . ( 2017 ) propose a joint multi-modal VAE ( JMVAE ) that is trained via two uni-modal encoders and a bimodal en-/decoder which share one objective function derived from the variation of information ( VI ) of the marginal conditional log-likelihoods log p ( a|b ) p ( b|a ) = : LM by optimizing the ELBO LM : LM ≥ LM ≥ LJ −DKL ( qφab ( z|a , b ) ‖qφb ( z|b ) ) ︸ ︷︷ ︸ Unimodal PDF fitting of encoder b −DKL ( qφab ( z|a , b ) ‖qφa ( z|a ) ) ︸ ︷︷ ︸ Unimodal PDF fitting of encoder a ( 3 ) Therefore , uni-modal encoders are trained , so that their distributions qφa and qφb are close to a multimodal encoder qφab in order to build a coherent posterior distribution . The introduced regularization by Suzuki et al . ( 2017 ) puts learning pressure on the uni-modal encoders just by the distributions ’ shape , disregarding reconstruction capabilities and the prior p ( z ) . Furthermore , one can show that deriving the ELBO from the VI for a set of M observable modalities , always leads to an expression of the ELBO that allows only training of M̃ = { m|m ∈ P ( M ) , |m| = |M| − 1 } modality combinations . This leads to the fact that for instance in a tri-modal setup , as shown in Fig . 1 ( right ) , one can derive three bi-modal encoders from the VI , but no uni-modal ones . 3 MULTI-MODAL VARIATIONAL AUTO ENCODER APPROACH . While the objective of Wang et al . ( 2016 ) , Ngiam et al . ( 2011 ) , Suzuki et al . ( 2017 ) , and Vedantam et al . ( 2017 ) is to exchange modalities bi-directionally ( e.g . a → b′ ) , our primary concern is twofold : First , find a meaningful posterior distribution where the sampled statistics of an encoder network allows inference about further actions . Second , find an expression to jointly train all 2|M|−1 permutations of modality encoders . By successively applying logarithm and Bayes rules , we derive the ELBO for the multi-modal VAE ( M2VAE ) as follows : First , given the independent set of observable modalities M = { a , b , c , . . . } , its marginal log-likelihood log p ( M ) = : LM2 is multiplied by the cardinality of the set as the neutral element 1 = |M|/|M| . Second , applying logarithm multiplication rule , the nominator is written as the argument ’ s exponent . Third , Bayes rule is applied to each term wrt . the remaining observable modalities to derive their conditionals . Further , we bootstrap the derivation technique in a bi- and tri-modal ( c.f . tri-modal case in Sec . 6.1 ) case to illustrate the advantages . By excessively applying the scheme until convergence of the mathematical expression , it leads for a bi-modal set M = { a , b } to the following result : LM2 = 2/2 log p ( a , b ) = 1/2 log p ( a , b ) 2 = 1/2 log p ( a , b ) p ( a , b ) = 1/2 log p ( b ) p ( a|b ) p ( b|a ) p ( a ) ( 4 ) = 1/2 ( log p ( a ) + log p ( b|a ) + log p ( a|b ) + log p ( b ) ) = 1/2 ( La + LM + Lb ) ( 5 ) This term can be written as inequality wrt . each ELBO of the marginals La , Lb and conditionals LM : 2LM2 ≥ 2LM2 = La + Lb + LM = ( 6 ) − βa DKL ( qφa ( z|a ) ‖p ( z ) ) + Eqφa ( z|a ) log ( pθa ( a|z ) ) ( 7 ) − βb DKL ( qφb ( z|b ) ‖p ( z ) ) + Eqφb ( z|b ) log ( pθb ( b|z ) ) ( 8 ) + Eqφab ( z|a , b ) log ( pθa ( a|z ) ) + Eqφab ( z|a , b ) log ( pθb ( b|z ) ) − βab DKL ( qφab ( z|a , b ) ‖p ( z ) ) ( 9 ) − αDKL ( qφab ( z|a , b ) ‖qφa ( z|a ) ) − αDKL ( qφab ( z|a , b ) ‖qφb ( z|b ) ) . ( 10 ) Equation 6 is substituted by all formerly derived ELBO expressions lead to the combination of the uni-modal VAEs wrt . a and b ( c.f . Eq . 7 to 8 ) and the JMVAE comprising the VAE wrt . the joint modality ab ( c.f . Eq . 9 ) and mutual latent space ( c.f . Eq . 10 ) . Equation 7 and 8 have the effect that their regularizers care about the uni-modal distribution to deviate not too much from the common prior while their reconstruction term shapes the underlying embedding of the mutual latent space . We further apply the concept of β-VAE ( Higgins et al . ( 2016 ; 2017a ) ; Burgess et al . ( 2018 ) ) to the regularizers via β∗ and adopt the factor α from Suzuki et al . ( 2017 ) for the mutual regularizer . However , while β-VAE have the property to disentangle the latent space , our main concern is the balance between the input and the latent space using a constant normalized factor βnorm = β∗D∗/Dz . If the derivation , which we leave out for the sake of brevity , is applied to the log-likelihood LM2M of a set M , one can show that it results into a recursive form consisting of JMVAEs ’ and M2VAEs ’ log-likelihood terms LM2M = 1 |M| LMM + ∑ m̃∈M̃ LM2m̃ ≥ 1 |M| LMM + ∑ m̃∈M̃ LM2m̃ = : LM2M . ( 11 ) While the derivation of Eq . 11 is given in Sec 6.1.3 , the properties are as follows : • the M2VAE consist out of 2|M|−1 encoders and |M| decoders comprising all modality combinations • while it also allows the bi-directional exchange of modalities , it further allows the setup of arbitrary modality combinations having 1 to |M| modalities • subsets of minor cardinality are weighted less and have a therefore minor impact in shaping the overall posterior distribution ( vice versa , the major subsets dominate the shaping and the minor sets adapt to it ) • all encoder/decoder networks can jointly be trained using SGVB

Summary: This paper proposes an objective, M^2VAE, for multi-modal VAEs, which is supposed to learn a more meaningful latent space representation. To summarize my understanding of the proposed objective, in the bi-modal case, it combines both objectives of TELBO [1] and JMVAE-kl [2] with some hyperparameters to learn the uni-modal encoders. The terms of Eqns 7,8, and 9 are equivalent to TELBO and Eqns 9 and 10 are JMVAE-kl. It would be very beneficial for the readers if you could more clearly contrast your objective with the related work given how similar they are.