Learning Factorized Multimodal Representations

1 INTRODUCTION . Multimodal machine learning involves learning from data across multiple modalities ( Baltrušaitis et al. , 2017 ) . It is a challenging yet crucial research area with real-world applications in robotics ( Liu et al. , 2017 ) , dialogue systems ( Pittermann et al. , 2010 ) , intelligent tutoring systems ( Petrovica et al. , 2017 ) , and healthcare diagnosis ( Frantzidis et al. , 2010 ) . At the heart of many multimodal modeling tasks lies the challenge of learning rich representations from multiple modalities . For example , analyzing multimedia content requires learning multimodal representations across the language , visual , and acoustic modalities ( Cho et al. , 2015 ) . Although the presence of multiple modalities provides additional valuable information , there are two key challenges to address when learning from multimodal data : 1 ) models must learn the complex intra-modal and cross-modal interactions for prediction ( Zadeh et al. , 2017 ) , and 2 ) trained models must be robust to unexpected missing or noisy modalities during testing ( Ngiam et al. , 2011 ) . In this paper , we propose to optimize for a joint generative-discriminative objective across multimodal data and labels . The discriminative objective ensures that the representations learned are rich in intra-modal and cross-modal features useful towards predicting the label , while the generative objective allows the model to infer missing modalities at test time and deal with the presence of noisy modalities . To this end , we introduce the Multimodal Factorization Model ( MFM in Figure 1 ) that factorizes multimodal representations into multimodal discriminative factors and modality-specific generative factors . Multimodal discriminative factors are shared across all modalities and contain joint multimodal features required for discriminative tasks . Modality-specific generative factors are unique for each modality and contain the information required for generating each modality . We believe that factorizing multimodal representations into different explanatory factors can help each factor focus on learning from a subset of the joint information across multimodal data and labels . This method is in contrast to jointly learning a single factor that summarizes all generative and ∗equal contributions discriminative information ( Srivastava & Salakhutdinov , 2012 ) . To sum up , MFM defines a joint distribution over multimodal data , and by the conditional independence assumptions in the assumed graphical model , both generative and discriminative aspects are taken into account . Our model design further provides interpretability of the factorized representations . Through an extensive set of experiments , we show that MFM learns improved multimodal representations with these characteristics : 1 ) The multimodal discriminative factors achieve state-of-the-art or competitive performance on six multimodal time series datasets . We also demonstrate that MFM can generalize by integrating it with other existing multimodal discriminative models . 2 ) MFM allows flexible generation concerning multimodal discriminative factors ( labels ) and modality-specific generative factors ( styles ) . We further show that we can perform reconstruction of missing modalities from observed modalities without significantly impacting discriminative performance . Finally , we interpret our learned representations using information-based and gradient-based methods , allowing us to understand the contributions of individual factors towards multimodal prediction and generation . 2 MULTIMODAL FACTORIZATION MODEL . Multimodal Factorization Model ( MFM ) is a latent variable model ( Figure 1 ( a ) ) with conditional independence assumptions over multimodal discriminative factors and modality-specific generative factors . According to these assumptions , we propose a factorization over the joint distribution of multimodal data ( Section 2.1 ) . Since exact posterior inference on this factorized distribution can be intractable , we propose an approximate inference algorithm based on minimizing a joint-distribution Wasserstein distance over multimodal data ( Section 2.2 ) . Finally , we derive the MFM objective by approximating the joint-distribution Wasserstein distance via a generalized mean-field assumption . Notation : We define X1∶M as the multimodal data from M modalities and Y as the labels , with joint distribution PX1∶M , Y = P ( X1∶M , Y ) . Let X̂1∶M denote the generated multimodal data and Ŷ denote the generated labels , with joint distribution PX̂1∶M , Ŷ = P ( X̂1∶M , Ŷ ) . 2.1 FACTORIZED MULTIMODAL REPRESENTATIONS . To factorize multimodal representations into multimodal discriminative factors and modality-specific generative factors , MFM assumes a Bayesian network structure as shown in Figure 1 ( a ) . In this graphical model , factors Fy and Fa { 1∶M } are generated from mutually independent latent variables Z = [ Zy , Za { 1∶M } ] with prior PZ . In particular , Zy generates the multimodal discriminative factor Fy and Za { 1∶M } generate modality-specific generative factors Fa { 1∶M } . By construction , Fy contributes to the generation of Ŷ while { Fy , Fai } both contribute to the generation of X̂i . As a result , the joint distribution P ( X̂1∶M , Ŷ ) can be factorized as follows : P ( X̂1∶M , Ŷ ) = ∫ F , Z P ( X̂1∶M , Ŷ∣F ) P ( F∣Z ) P ( Z ) dFdZ = ∫Fy , Fa { 1∶M } Zy , Za { 1∶M } ( P ( Ŷ∣Fy ) M ∏ i=1 P ( X̂i∣Fai , Fy ) ) ( P ( Fy ∣Zy ) M ∏ i=1 P ( Fai∣Zai ) ) ( P ( Zy ) M ∏ i=1 P ( Zai ) ) dFdZ , ( 1 ) with dF = dFy∏Mi=1 dFai and dZ = dZy∏ M i=1 dZai . Exact posterior inference in Equation 1 may be analytically intractable due to the integration over Z . We therefore resort to using an approximate inference distribution Q ( Z∣X1∶M , Y ) as detailed in the following subsection . As a result , MFM can be viewed as an autoencoding structure that consists of encoder ( inference ) and decoder ( generative ) modules ( Figure 1 ( c ) ) . The encoder module for Q ( ⋅∣⋅ ) allows us to easily sample Z from an approximate posterior . The decoder modules are parametrized according to the factorization of P ( X̂1∶M , Ŷ∣Z ) as given by Equation 1 and Figure 1 ( a ) . 2.2 MINIMIZING JOINT-DISTRIBUTION WASSERSTEIN DISTANCE OVER MULTIMODAL DATA . Two common choices for approximate inference in autoencoding structures are Variational Autoencoders ( VAEs ) ( Kingma & Welling , 2013 ) and Wasserstein Autoencoders ( WAEs ) ( Zhao et al. , 2017 ; Tolstikhin et al. , 2017 ) . The former optimizes the evidence lower bound objective ( ELBO ) , and the latter derives an approximation for the primal form of the Wasserstein distance . We consider the latter since it simultaneously results in better latent factor disentanglement ( Zhao et al. , 2017 ; Rubenstein et al. , 2018 ) and better sample generation quality than its counterparts ( Chen et al. , 2016 ; Higgins et al. , 2016 ; Kingma & Welling , 2013 ) . However , WAEs are designed for unimodal data and do not consider factorized distributions over latent variables that generate multimodal data . Therefore , we propose a variant for handling factorized joint distributions over multimodal data . As suggested by Kingma & Welling ( 2013 ) , we adopt the design of nonlinear mappings ( i.e . neural network architectures ) in the encoder and decoder ( Figure 1 ( c ) ) . For the encoder Q ( Z∣X1∶M , Y ) , we learn a deterministic mapping Qenc ∶X1∶M , Y → Z ( Rubenstein et al. , 2018 ; Tolstikhin et al. , 2017 ) . For the decoder , we define the generation process from latent variables as Gy ∶ Zy → Fy , Ga { 1∶M } ∶ Za { 1∶M } → Fa { 1∶M } , D ∶ Fy → Ŷ , and F1∶M ∶ Fy , Fa { 1∶M } → X̂1∶M , where Gy , Ga { 1∶M } , D and F1∶M are deterministic functions parametrized by neural networks . Let Wc ( PX1∶M , Y , PX̂1∶M , Ŷ ) denote the joint-distribution Wasserstein distance over multimodal data under cost function cXi and cY . We choose the squared cost c ( a , b ) = ∥a − b∥22 , allowing us to minimize the 2-Wasserstein distance . The cost function can be defined not only on static data but also on time series data such as text , audio and videos . For example , given time series data X = [ X1 , X2 , ⋯ , XT ] and X̂ = [ X̂1 , X̂2 , ⋯ , X̂T ] , we define c ( X , X̂ ) = ∑Tt=1 ∥Xt − X̂t∥22 . With conditional independence assumptions in Equation 1 , we express Wc ( PX1∶M , Y , PX̂1∶M , Ŷ ) as : Proposition 1 . For any functions Gy ∶ Zy → Fy , Ga { 1∶M } ∶ Za { 1∶M } → Fa { 1∶M } , D ∶ Fy → Ŷ , and F1∶M ∶ Fa { 1∶M } , Fy → X̂1∶M , we have Wc ( PX1∶M , Y , PX̂1∶M , Ŷ ) = inf QZ=PZ EPX1∶M , YEQ ( Z∣X1∶M , Y ) ⎡⎢⎢⎢⎢⎣ M ∑ i=1 cXi ( Xi , Fi ( Gai ( Zai ) , Gy ( Zy ) ) ) + cY ( Y , D ( Gy ( Zy ) ) ) ⎤⎥⎥⎥⎥⎦ , ( 2 ) where PZ is the prior over Z = [ Zy , Za { 1 , M } ] and QZ is the aggregated posterior of the proposed approximate inference distribution Q ( Z∣X1∶M , Y ) . Proof : The proof is adapted from Tolstikhin et al . ( Tolstikhin et al. , 2017 ) . The two differences are : ( 1 ) we show that P ( X̂1∶M , Ŷ∣Z = z ) are Dirac for all z ∈ Z , and ( 2 ) we use the fact that c ( ( X1∶M , Y ) , ( X̂1∶M , Ŷ ) ) = ∑Mi=1 cXi ( Xi , X̂i ) + cY ( Y , Ŷ ) . Please refer to the supplementary material for proof details . ∎ The constraint on QZ = PZ in Proposition 1 is hard to satisfy . To obtain a numerical solution , we first relax the constraint by performing a generalized mean field assumption on Q according to the conditional independence as shown in the inference network of Figure 1 ( b ) : Q ( Z∣X1∶M , Y ) ∶= Q ( Z∣X1∶M ) ∶= Q ( Zy∣X1∶M ) M ∏ i=1 Q ( Zai∣Xi ) . ( 3 ) The intuition here is based on our design that Zy generates the multimodal discriminative factor Fy and Za { 1∶M } generate modality-specific generative factors Fa { 1∶M } . Therefore , the inference for Zy should depend on all modalities X1∶M and the inference for Zai should depend only on the specific modality Xi . Following this assumption , we define Q as a nonparametric set of all encoders that fulfill the factorization in Equation 3 . A penalty term is added into our objective to find the Q ( Z∣⋅ ) ∈ Q that is the closest to prior PZ , thereby approximately enforcing the constraint QZ = PZ : min F , Ga { 1∶M } , Gy , D inf Q ( Z∣⋅ ) ∈Q EPX1∶M , YEQ ( Za1 ∣X1 ) ⋯EQ ( ZaM ∣XM ) EQ ( Zy ∣X1∶M ) ⎡⎢⎢⎢⎢⎣ M ∑ i=1 cXi ( Xi , F ( Gai ( Zai ) , Gy ( Zy ) ) ) + cY ( Y , D ( Gy ( Zy ) ) ) ⎤⎥⎥⎥⎥⎦ + λMMD ( QZ , PZ ) , ( 4 ) where λ is a hyper-parameter andMMD is the Maximum Mean Discrepancy ( Gretton et al. , 2012 ) as a divergence measure between QZ and PZ . The prior PZ is chosen as a centered isotropic Gaussian N ( 0 , I ) , so that it implicitly enforces independence between the latent variables Z = [ Zy , Za { 1 , M } ] ( Higgins et al. , 2016 ; Kingma & Welling , 2013 ; Rubenstein et al. , 2018 ) . Equation 4 represents our hybrid generative-discriminative optimization objective over multimodal data : the first loss term ∑Mi=1 cXi ( Xi , F ( Gai ( Zai ) , Gy ( Zy ) ) ) is the generative objective based on reconstruction of multimodal data and the second term cY ( Y , D ( Gy ( Zy ) ) ) is the discriminative objective . In practice we compute the expectations in Equation 4 using empirical estimates over the training data . The neural architecture of MFM is illustrated in Figure 1 ( c ) .

Summary: Multimodality learning is an important topic in multimedia and human computer interaction.  How to efficiently leverage the additional information cross multimodality is the key to the task. Authors proposed the Bayesian latent variable model to factorize the multimodality representation into multimodal discriminative factors and modality-specific generating factors, which is interesting. Approximate inference is also proposed to learn this model via a generalised mean-field assumption.