Importance-based Multimodal Autoencoder

Integrating information from multiple modalities ( e.g. , verbal , acoustic and visual data ) into meaningful representations has seen great progress in recent years . However , two challenges are not sufficiently addressed by current approaches : ( 1 ) computationally efficient training of multimodal autoencoder networks which are robust in the absence of modalities , and ( 2 ) unsupervised learning of important subspaces in each modality which are correlated with other modalities . In this paper we propose the IMA ( Importance-based Multimodal Autoencoder ) model , a scalable model that learns modality importances and robust multimodal representations through a novel cross-covariance based loss function . We conduct experiments on MNIST-TIDIGITS , a multimodal dataset of spoken and image digits , and on IEMOCAP , a multimodal emotion corpus . The IMA model is able to distinguish digits from uncorrelated noise , and word-level importances are learnt that correspond to the separation between function and emotional words . The multimodal representations learnt by IMA are also competitive with state-of-the-art baseline approaches on downstream tasks . 1 INTRODUCTION . With the ever-increasing amount of heterogeneous multimedia content present on the internet , machine learning approaches have been applied to automated perception problems such as object recognition ( Krizhevsky et al. , 2012 ) , image captioning ( Vinyals et al. , 2015 ) and automatic language translation ( Choi et al. , 2018 ) . An important research direction is the problem of learning representations from multiple modalities which represent our primary channels of communication and sensation , such as vision or touch ( Baltrušaitis et al. , 2018 ) . With respect to this area of research , there are two major challenges in this research area which our paper addresses . The first is the design of encoder networks to enable learning and inference of multimodal representations in the absence of modalities . This is useful for scenarios such as sensor failure or imputation/bidirectional generation of missing modalities from any combination of the observed ones . The caveat is that to have this property , recent approaches such as the JMVAE-KL model ( Suzuki et al. , 2016 ) and MVAE ( Wu & Goodman , 2018 ) have encoders with high complexity for a large number of modalities . When M is the number of modalities , JMVAE-KL needs 2M sub-networks for every combination of input modalities , while MVAE requires M sub-networks but additional O ( 2M ) subsampled loss terms to handle missing modalities . The second challenge is that multimodal data , such as emotional spoken utterances or web images with captions are often generated not only by an underlying shared latent factor , but also by modality-specific private latent factors . For example in spoken utterances , the verbal modality ( words ) are generated not only due to emotion but also due to syntax and semantics . Function words such as I and the are mostly syntactic and do not relate to emotion , similarly not all recorded audio frames are indicative of emotion . The inference of how relevant a sample in each modality is to the shared latent factor ( subsequently referred to as importance ) is important for downstream tasks . For the remainder of this paper , non-relevant samples are refered to as uncorrelated noise . In a supervised setting , the latent factors and modality importance weights can be learnt from task labels . When labels are absent in the unsupervised scenario , for the purpose of this paper we define the concept of modality importances based on correlations between the latent factor and each modality . In the important subspace of each modality the multimodal and unimodal representations both maximally correlate , indicating that samples in that modality subspace can be attributed to a shared latent factor , and not an independent private one . In contrast , for unimportant samples in a modality , the correlation is minimal . The main contributions of our proposed approach are two-fold . The first is a multimodal autoencoder framework where training requires additional loss terms which are O ( M ) , i.e . linear in the number of modalities , and thus only require M per-modality encoders to handle missing modalities . Computationally , this is advantageous compared to JMVAE-KL and MVAE , which require exponential number of sub-networks and loss terms respectively . Secondly , we define the concept of importance in an unsupervised setting , and propose novel cross-correlation based loss terms to learn important regions in each modality ’ s representation space . The importances are modeled by separate unimodal networks referred to as importance networks . Hyper-parameter ρj for the j-th modality controls the integration of prior domain knowledge about the degree of importance in that modality . While not trained on any supervised labels , the learnt importances from IMA are analyzed quantitatively and found to correspond to the separation between digit vs. noise labels and emotion vs. neutral categories . 2 RELATED WORK . Following the great success of deep neural networks for representation learning , the research area of multimodal machine learning is gaining interest ( Baltrušaitis et al. , 2018 ) . Our proposed IMA model is relevant to two main research areas in this domain , Inter-modality Correlation Learning and Efficient Multimodal VAEs . The idea of learning acoustic embeddings for words has also been explored in Wang et al . ( 2018 ) and Jung et al . ( 2019 ) however we attempt to map words to their affective rather than phonetic representations . In this section , we describe each area and conclude with the similarities and differences between the IMA model and prior approaches . Inter-modality Correlation Learning : There have been several approaches which measure correlations between modalities/sources of data to understand how observed data in each modality can be explained by shared underlying concepts . The IBFA ( Inter-Battery Factor Analysis ) introduced by Tucker ( 1958 ) and its successor , the MBFA ( Multi-battery factor Analysis ) ( Browne , 1980 ) are among the earliest proposed techniques to study shared factors between score sets from batteries of tests . DeepCCA ( Deep Canonical Covariance Analysis ) proposed by Benton et al . ( 2017 ) learns a deep projection of each modality in a bimodal understanding scenario so that the projections are maximally correlated , effectively extending the classical CCA technique ( Knapp , 1978 ) to deep neural networks . Our proposed model extends these approaches to also detect important regions of each modality correlated with the shared latent factor . Efficient Multimodal VAEs : VAEs ( Variational Autoencoders ) have been applied to multimodal data for applications such as inference and bidirectional generation of modalities . This poses a major challenge of constructing encoders to model the latent posterior which are efficient in training/inference under any combination of input modalities . Recent work addresses this by focusing on factorized models for efficient inference . Vedantam et al . ( 2017 ) employs a product-of-experts decomposition with modality specific inference networks to train image generation models . Wu & Goodman ( 2018 ) propose MVAE ( Multimodal Variational Autoencoders ) , where the latent posterior is modeled with a parameter shared product of experts network . Shi et al . ( 2019 ) proposed a mixture-of-experts multimodal variational autoencoder ( MMVAE ) where the posterior is a mixture of experts instead . These approaches have been extended more recently , for example in multi-source neural variational inference ( Kurle et al. , 2019 ) where the multimodal posterior is constructed using learnt and integrated beliefs from multiple posteriors , each being informed by a different source . Sutter et al . ( 2020 ) introduce a novel Jensen-Shannon divergence based objective function which can be used to approximate both unimodal and joint multimodal posteriors . While existing approaches attempt to efficiently learn multimodal representations through posterior modeling , our proposed IMA model aligns modalities during autoencoder training for projection to a common space which facilitates inference even in absence of modalities . Only M encoders and O ( M ) loss terms are required by the IMA model for inference with M modalities . Prior work has also not focused sufficiently on unsupervised learning of modality importances ( through detection of subspaces maximally correlated with shared latent factors ) which we address in this paper . 3 MODEL DESCRIPTION . The IMA model consists of two main components : ( 1 ) the multimodal autoencoder and ( 2 ) the unimodal importance networks . In Figure 1 we have provided an overview diagram of the proposed model , including the loss functions utilized in training . Multimodal Autoencoder : Assume that the input training examples consist of multimodal data , where each multimodal sample is denoted as x = { x1 , x2 , ... xM } . There are M modalities and the training set consists of N multimodal samples . The input data xj in the j-th modality is passed through an encoder for that modality and its output is denoted as uj ( xj ) . The latent multimodal representation z could be modeled with different approaches for example , concatenated fusion with fully connected layers ( Suzuki et al. , 2016 ) . We wish to model the latent multimodal representation z ( x1 , x2 ... xM ) with a pooling weighted by distinct modality importances yj ( xj ) as given by : z ( x1 , x2 , ... xM ) = M∑ j=1 yj ( xj ) uj ( xj ) ∑ j yj ( xj ) = 1 ( 1 ) The multimodal representation z is passed through the decoder networks to obtain the reconstruction in the j-th modality as x̂j . L ( j ) rec is the reconstruction loss for the j-th modality . Lglob is a global regularization term which encourages z to be centered with zero mean . This term is for regularization of the multimodal representation z and expands to Lglob ( z ) = ‖z ( x1 , x2 , ... xM ) ‖2 . Multimodal alignment in IMA : For our proposed model , we have M sub-networks for each modality , where each j-th unimodal sub-network is trained using the autoencoder reconstruction losses and an additional alignment loss L ( j ) align . By forcing the multimodal representation z and each modality ’ s view uj ( xj ) to be similar , we also enforce each unimodal sub-network to learn its contribution to the latent factor z during autoencoder training . Per-sample , this requires only M sub-networks and M additional loss terms instead of a random subset of 2M losses for sub-sampled training as in the MVAE model ( Wu & Goodman , 2018 ) or 2M subnetworks as in the JMVAE-KL model ( Suzuki et al. , 2016 ) . Similar to MVAE , this model also can learn and infer in the absence of modalities . z can be expressed as a linear average of the unimodal representations present and incorporated into the overall loss . If the j-th modality is missing , we can set yj ( xj ) = 0 in Equation 1 . For the j-th modality , L ( j ) align is the SSE error between the multimodal representation z and the unimodal encoder outputs uj ( xj ) given by : L ( j ) align ( z , uj ) = ‖z ( x1 , x2 , ... xM ) − uj ( xj ) ‖ 2 When there are N training samples , during multimodal autoencoder learning the loss below is optimized : Lauto = λglob N∑ i=1 Lglob ( zi ) + [ N∑ i=1 M∑ j=1 λ ( j ) recL ( j ) rec ( xij , x̂ij ) ] + [ N∑ i=1 M∑ j=1 λ ( j ) alignL ( j ) align ( zi , uij ) ] ( 2 ) λalign , λrec and λglob are the associated hyper-parameter weights for the alignment , reconstruction and global multimodal regularization terms which appear in Equation ( 2 ) . Since it is not straightforward to tune such parameters for an unsupervised model , we start with equal weights for all losses and observe if that is sufficient to minimize all terms simultaneously during training . For the MNIST-TIDIGITS and IEMOCAP experiments , we have found that equally weighing them works , thus λalign = λrec = λglob = 1.0 . Importance Network Training : Section 1 explains that there are samples in observed data for each modality which do not correlate with the multimodal representation z . For the scope of this paper , we consider such samples as uncorrelated noise in the data . We have assumed the presence of a subspace Rj inside each j-th modality ’ s representation space so that if xj ∈ Rj , uj is minimally correlated with z . We seek to learn weights yij ∈ [ 0 , 1 ] for the i-th sample in the j−th modality , so that yij denotes the importance of each sample xij ( i.e . the degree to which xij does not belong to Rj ) . A unimodal neural network is trained to map from xij to yij for the j-th modality ; this is called the importance network . The multimodal autoencoder solely being trained to reconstruct all modality inputs would not have learnt yij corresponding to uncorrelated noise . This motivates the need for explicitly defining an importance-based loss . Cross-correlation Losses : To train the importance networks , we have made use of a loss function explicitly capturing cross-correlation between z and uj , weighted by yj . Minimizing this cost is equivalent to enforcing zero-correlation between uj and z based on mini-batch statistics during training , where size of a mini-batch is B samples . This cost can be derived from the definition of cross-correlation , where each i-th sample in the mini-batch is not weighted equally . Recent work has also applied other independence criterion , such as MMD ( Maximum Mean Discrepancies ) which act as auxiliary losses in a variational framework ( Louizos et al. , 2015 ) . Assuming the latent variables are Gaussians , independence and uncorrelated properties are equivalent , and we define the following alternative loss term based on the Frobenius norm of the cross-covariance between z and uj , where y′ij = 1− yij and w′ij = y′ij / B∑ i=1 y′ij L ( j ) corr = ∥∥∥∥∥ B∑ i=1 w′ij [ ( z− µz ) ( uj − µu ) ] T ∥∥∥∥∥ 2 F µz = B∑ i=1 w′ijzi µu = B∑ i=1 w′ijuij ( 3 ) Importance Priors : Learning to predict yij ≈ 1.0 ∀i , j trivially decreases Lcorr down to zero , and thus we need to regularize the importance network training , through an additional loss function which we refer to as L ( j ) local . This loss utilizes the hyper-parameter ρj ∈ [ 0 , 1 ] which serves as prior about how much of the j-th modality is corrupted by uncorrelated noise , and is defined as the KL-divergence DKL ( Bernoulli ( ȳj ) ||Bernoulli ( ρj ) ) where ȳj = ∑ i yij/B is the average value of yij as computed over a mini-batch of size B. Enforcing the loss Llocal at a sample level would have forced yij = ρj for all samples , which is addressed by defining the loss on a mini-batch instead . The importance network for each modality minimizes Limp , which is the weighted sum of the cross-covariance based loss and the regularization term , as defined below . λ ( j ) local and λ ( j ) corr are hyper-parameter weights for each loss term . Limp = N∑ i=1 M∑ j=1 λ ( j ) localL ( j ) local + N∑ i=1 M∑ j=1 λ ( j ) corrL ( j ) corr ( 4 ) It is important to note the difference between yij and y′ij = 1 − yij in terms of notation . yij is the importance of the i−th sample in the j−th modality and is required in the joint representation , whereas its complement y′ij is utilized in importance network training . The intuition is that a model minimizing Equation 3 would tend to learn high values of y′ij for maximally uncorrelated samples , which translates to low importance ( yij ) for these same samples .

Summary: This paper presents a multimodal Autoencoder framework that learns the multimodal latent representations alongwith the importance of regions in each modality’s representation space in an unsupervised fashion. Multimodal fusion algorithms either use complex architecture representations or use disentangling joint representations for improving generative auto-encoding architectures using VAEs, GANs, WAE and some variants of these. This paper presents an elegant importance based model and architecture that takes into account various local and joint loss functions along with alignment factors to represent the Autoencoder model.