Learning Factorized Multimodal Representations

1 INTRODUCTION . Multimodal machine learning involves learning from data across multiple modalities ( Baltrušaitis et al. , 2017 ) . It is a challenging yet crucial research area with real-world applications in robotics ( Liu et al. , 2017 ) , dialogue systems ( Pittermann et al. , 2010 ) , intelligent tutoring systems ( Petrovica et al. , 2017 ) , and healthcare diagnosis ( Frantzidis et al. , 2010 ) . At the heart of many multimodal modeling tasks lies the challenge of learning rich representations from multiple modalities . For example , analyzing multimedia content requires learning multimodal representations across the language , visual , and acoustic modalities ( Cho et al. , 2015 ) . Although the presence of multiple modalities provides additional valuable information , there are two key challenges to address when learning from multimodal data : 1 ) models must learn the complex intra-modal and cross-modal interactions for prediction ( Zadeh et al. , 2017 ) , and 2 ) trained models must be robust to unexpected missing or noisy modalities during testing ( Ngiam et al. , 2011 ) . In this paper , we propose to optimize for a joint generative-discriminative objective across multimodal data and labels . The discriminative objective ensures that the representations learned are rich in intra-modal and cross-modal features useful towards predicting the label , while the generative objective allows the model to infer missing modalities at test time and deal with the presence of noisy modalities . To this end , we introduce the Multimodal Factorization Model ( MFM in Figure 1 ) that factorizes multimodal representations into multimodal discriminative factors and modality-specific generative factors . Multimodal discriminative factors are shared across all modalities and contain joint multimodal features required for discriminative tasks . Modality-specific generative factors are unique for each modality and contain the information required for generating each modality . We believe that factorizing multimodal representations into different explanatory factors can help each factor focus on learning from a subset of the joint information across multimodal data and labels . This method is in contrast to jointly learning a single factor that summarizes all generative and ∗equal contributions discriminative information ( Srivastava & Salakhutdinov , 2012 ) . To sum up , MFM defines a joint distribution over multimodal data , and by the conditional independence assumptions in the assumed graphical model , both generative and discriminative aspects are taken into account . Our model design further provides interpretability of the factorized representations . Through an extensive set of experiments , we show that MFM learns improved multimodal representations with these characteristics : 1 ) The multimodal discriminative factors achieve state-of-the-art or competitive performance on six multimodal time series datasets . We also demonstrate that MFM can generalize by integrating it with other existing multimodal discriminative models . 2 ) MFM allows flexible generation concerning multimodal discriminative factors ( labels ) and modality-specific generative factors ( styles ) . We further show that we can perform reconstruction of missing modalities from observed modalities without significantly impacting discriminative performance . Finally , we interpret our learned representations using information-based and gradient-based methods , allowing us to understand the contributions of individual factors towards multimodal prediction and generation . 2 MULTIMODAL FACTORIZATION MODEL . Multimodal Factorization Model ( MFM ) is a latent variable model ( Figure 1 ( a ) ) with conditional independence assumptions over multimodal discriminative factors and modality-specific generative factors . According to these assumptions , we propose a factorization over the joint distribution of multimodal data ( Section 2.1 ) . Since exact posterior inference on this factorized distribution can be intractable , we propose an approximate inference algorithm based on minimizing a joint-distribution Wasserstein distance over multimodal data ( Section 2.2 ) . Finally , we derive the MFM objective by approximating the joint-distribution Wasserstein distance via a generalized mean-field assumption . Notation : We define X1∶M as the multimodal data from M modalities and Y as the labels , with joint distribution PX1∶M , Y = P ( X1∶M , Y ) . Let X̂1∶M denote the generated multimodal data and Ŷ denote the generated labels , with joint distribution PX̂1∶M , Ŷ = P ( X̂1∶M , Ŷ ) . 2.1 FACTORIZED MULTIMODAL REPRESENTATIONS . To factorize multimodal representations into multimodal discriminative factors and modality-specific generative factors , MFM assumes a Bayesian network structure as shown in Figure 1 ( a ) . In this graphical model , factors Fy and Fa { 1∶M } are generated from mutually independent latent variables Z = [ Zy , Za { 1∶M } ] with prior PZ . In particular , Zy generates the multimodal discriminative factor Fy and Za { 1∶M } generate modality-specific generative factors Fa { 1∶M } . By construction , Fy contributes to the generation of Ŷ while { Fy , Fai } both contribute to the generation of X̂i . As a result , the joint distribution P ( X̂1∶M , Ŷ ) can be factorized as follows : P ( X̂1∶M , Ŷ ) = ∫ F , Z P ( X̂1∶M , Ŷ∣F ) P ( F∣Z ) P ( Z ) dFdZ = ∫Fy , Fa { 1∶M } Zy , Za { 1∶M } ( P ( Ŷ∣Fy ) M ∏ i=1 P ( X̂i∣Fai , Fy ) ) ( P ( Fy ∣Zy ) M ∏ i=1 P ( Fai∣Zai ) ) ( P ( Zy ) M ∏ i=1 P ( Zai ) ) dFdZ , ( 1 ) with dF = dFy∏Mi=1 dFai and dZ = dZy∏ M i=1 dZai . Exact posterior inference in Equation 1 may be analytically intractable due to the integration over Z . We therefore resort to using an approximate inference distribution Q ( Z∣X1∶M , Y ) as detailed in the following subsection . As a result , MFM can be viewed as an autoencoding structure that consists of encoder ( inference ) and decoder ( generative ) modules ( Figure 1 ( c ) ) . The encoder module for Q ( ⋅∣⋅ ) allows us to easily sample Z from an approximate posterior . The decoder modules are parametrized according to the factorization of P ( X̂1∶M , Ŷ∣Z ) as given by Equation 1 and Figure 1 ( a ) . 2.2 MINIMIZING JOINT-DISTRIBUTION WASSERSTEIN DISTANCE OVER MULTIMODAL DATA . Two common choices for approximate inference in autoencoding structures are Variational Autoencoders ( VAEs ) ( Kingma & Welling , 2013 ) and Wasserstein Autoencoders ( WAEs ) ( Zhao et al. , 2017 ; Tolstikhin et al. , 2017 ) . The former optimizes the evidence lower bound objective ( ELBO ) , and the latter derives an approximation for the primal form of the Wasserstein distance . We consider the latter since it simultaneously results in better latent factor disentanglement ( Zhao et al. , 2017 ; Rubenstein et al. , 2018 ) and better sample generation quality than its counterparts ( Chen et al. , 2016 ; Higgins et al. , 2016 ; Kingma & Welling , 2013 ) . However , WAEs are designed for unimodal data and do not consider factorized distributions over latent variables that generate multimodal data . Therefore , we propose a variant for handling factorized joint distributions over multimodal data . As suggested by Kingma & Welling ( 2013 ) , we adopt the design of nonlinear mappings ( i.e . neural network architectures ) in the encoder and decoder ( Figure 1 ( c ) ) . For the encoder Q ( Z∣X1∶M , Y ) , we learn a deterministic mapping Qenc ∶X1∶M , Y → Z ( Rubenstein et al. , 2018 ; Tolstikhin et al. , 2017 ) . For the decoder , we define the generation process from latent variables as Gy ∶ Zy → Fy , Ga { 1∶M } ∶ Za { 1∶M } → Fa { 1∶M } , D ∶ Fy → Ŷ , and F1∶M ∶ Fy , Fa { 1∶M } → X̂1∶M , where Gy , Ga { 1∶M } , D and F1∶M are deterministic functions parametrized by neural networks . Let Wc ( PX1∶M , Y , PX̂1∶M , Ŷ ) denote the joint-distribution Wasserstein distance over multimodal data under cost function cXi and cY . We choose the squared cost c ( a , b ) = ∥a − b∥22 , allowing us to minimize the 2-Wasserstein distance . The cost function can be defined not only on static data but also on time series data such as text , audio and videos . For example , given time series data X = [ X1 , X2 , ⋯ , XT ] and X̂ = [ X̂1 , X̂2 , ⋯ , X̂T ] , we define c ( X , X̂ ) = ∑Tt=1 ∥Xt − X̂t∥22 . With conditional independence assumptions in Equation 1 , we express Wc ( PX1∶M , Y , PX̂1∶M , Ŷ ) as : Proposition 1 . For any functions Gy ∶ Zy → Fy , Ga { 1∶M } ∶ Za { 1∶M } → Fa { 1∶M } , D ∶ Fy → Ŷ , and F1∶M ∶ Fa { 1∶M } , Fy → X̂1∶M , we have Wc ( PX1∶M , Y , PX̂1∶M , Ŷ ) = inf QZ=PZ EPX1∶M , YEQ ( Z∣X1∶M , Y ) ⎡⎢⎢⎢⎢⎣ M ∑ i=1 cXi ( Xi , Fi ( Gai ( Zai ) , Gy ( Zy ) ) ) + cY ( Y , D ( Gy ( Zy ) ) ) ⎤⎥⎥⎥⎥⎦ , ( 2 ) where PZ is the prior over Z = [ Zy , Za { 1 , M } ] and QZ is the aggregated posterior of the proposed approximate inference distribution Q ( Z∣X1∶M , Y ) . Proof : The proof is adapted from Tolstikhin et al . ( Tolstikhin et al. , 2017 ) . The two differences are : ( 1 ) we show that P ( X̂1∶M , Ŷ∣Z = z ) are Dirac for all z ∈ Z , and ( 2 ) we use the fact that c ( ( X1∶M , Y ) , ( X̂1∶M , Ŷ ) ) = ∑Mi=1 cXi ( Xi , X̂i ) + cY ( Y , Ŷ ) . Please refer to the supplementary material for proof details . ∎ The constraint on QZ = PZ in Proposition 1 is hard to satisfy . To obtain a numerical solution , we first relax the constraint by performing a generalized mean field assumption on Q according to the conditional independence as shown in the inference network of Figure 1 ( b ) : Q ( Z∣X1∶M , Y ) ∶= Q ( Z∣X1∶M ) ∶= Q ( Zy∣X1∶M ) M ∏ i=1 Q ( Zai∣Xi ) . ( 3 ) The intuition here is based on our design that Zy generates the multimodal discriminative factor Fy and Za { 1∶M } generate modality-specific generative factors Fa { 1∶M } . Therefore , the inference for Zy should depend on all modalities X1∶M and the inference for Zai should depend only on the specific modality Xi . Following this assumption , we define Q as a nonparametric set of all encoders that fulfill the factorization in Equation 3 . A penalty term is added into our objective to find the Q ( Z∣⋅ ) ∈ Q that is the closest to prior PZ , thereby approximately enforcing the constraint QZ = PZ : min F , Ga { 1∶M } , Gy , D inf Q ( Z∣⋅ ) ∈Q EPX1∶M , YEQ ( Za1 ∣X1 ) ⋯EQ ( ZaM ∣XM ) EQ ( Zy ∣X1∶M ) ⎡⎢⎢⎢⎢⎣ M ∑ i=1 cXi ( Xi , F ( Gai ( Zai ) , Gy ( Zy ) ) ) + cY ( Y , D ( Gy ( Zy ) ) ) ⎤⎥⎥⎥⎥⎦ + λMMD ( QZ , PZ ) , ( 4 ) where λ is a hyper-parameter andMMD is the Maximum Mean Discrepancy ( Gretton et al. , 2012 ) as a divergence measure between QZ and PZ . The prior PZ is chosen as a centered isotropic Gaussian N ( 0 , I ) , so that it implicitly enforces independence between the latent variables Z = [ Zy , Za { 1 , M } ] ( Higgins et al. , 2016 ; Kingma & Welling , 2013 ; Rubenstein et al. , 2018 ) . Equation 4 represents our hybrid generative-discriminative optimization objective over multimodal data : the first loss term ∑Mi=1 cXi ( Xi , F ( Gai ( Zai ) , Gy ( Zy ) ) ) is the generative objective based on reconstruction of multimodal data and the second term cY ( Y , D ( Gy ( Zy ) ) ) is the discriminative objective . In practice we compute the expectations in Equation 4 using empirical estimates over the training data . The neural architecture of MFM is illustrated in Figure 1 ( c ) .

Summary: This paper presents 'Multimodal Factorization model' that factorizes representations into shared multimodal discriminative factors and modality specific generative factors. This work applies 'Wassertein Auto-Encoders' by Tolstikhin et al (with proofs that this setup works in the multimodal case) for handling factorized joint distributions over the multimodal space. Can this method be considered as a generalization of the wasserstein autoencoder based method with a broader application? - the authors should discuss this more broadly in the paper.