Importance-based Multimodal Autoencoder

Integrating information from multiple modalities ( e.g. , verbal , acoustic and visual data ) into meaningful representations has seen great progress in recent years . However , two challenges are not sufficiently addressed by current approaches : ( 1 ) computationally efficient training of multimodal autoencoder networks which are robust in the absence of modalities , and ( 2 ) unsupervised learning of important subspaces in each modality which are correlated with other modalities . In this paper we propose the IMA ( Importance-based Multimodal Autoencoder ) model , a scalable model that learns modality importances and robust multimodal representations through a novel cross-covariance based loss function . We conduct experiments on MNIST-TIDIGITS , a multimodal dataset of spoken and image digits , and on IEMOCAP , a multimodal emotion corpus . The IMA model is able to distinguish digits from uncorrelated noise , and word-level importances are learnt that correspond to the separation between function and emotional words . The multimodal representations learnt by IMA are also competitive with state-of-the-art baseline approaches on downstream tasks . 1 INTRODUCTION . With the ever-increasing amount of heterogeneous multimedia content present on the internet , machine learning approaches have been applied to automated perception problems such as object recognition ( Krizhevsky et al. , 2012 ) , image captioning ( Vinyals et al. , 2015 ) and automatic language translation ( Choi et al. , 2018 ) . An important research direction is the problem of learning representations from multiple modalities which represent our primary channels of communication and sensation , such as vision or touch ( Baltrušaitis et al. , 2018 ) . With respect to this area of research , there are two major challenges in this research area which our paper addresses . The first is the design of encoder networks to enable learning and inference of multimodal representations in the absence of modalities . This is useful for scenarios such as sensor failure or imputation/bidirectional generation of missing modalities from any combination of the observed ones . The caveat is that to have this property , recent approaches such as the JMVAE-KL model ( Suzuki et al. , 2016 ) and MVAE ( Wu & Goodman , 2018 ) have encoders with high complexity for a large number of modalities . When M is the number of modalities , JMVAE-KL needs 2M sub-networks for every combination of input modalities , while MVAE requires M sub-networks but additional O ( 2M ) subsampled loss terms to handle missing modalities . The second challenge is that multimodal data , such as emotional spoken utterances or web images with captions are often generated not only by an underlying shared latent factor , but also by modality-specific private latent factors . For example in spoken utterances , the verbal modality ( words ) are generated not only due to emotion but also due to syntax and semantics . Function words such as I and the are mostly syntactic and do not relate to emotion , similarly not all recorded audio frames are indicative of emotion . The inference of how relevant a sample in each modality is to the shared latent factor ( subsequently referred to as importance ) is important for downstream tasks . For the remainder of this paper , non-relevant samples are refered to as uncorrelated noise . In a supervised setting , the latent factors and modality importance weights can be learnt from task labels . When labels are absent in the unsupervised scenario , for the purpose of this paper we define the concept of modality importances based on correlations between the latent factor and each modality . In the important subspace of each modality the multimodal and unimodal representations both maximally correlate , indicating that samples in that modality subspace can be attributed to a shared latent factor , and not an independent private one . In contrast , for unimportant samples in a modality , the correlation is minimal . The main contributions of our proposed approach are two-fold . The first is a multimodal autoencoder framework where training requires additional loss terms which are O ( M ) , i.e . linear in the number of modalities , and thus only require M per-modality encoders to handle missing modalities . Computationally , this is advantageous compared to JMVAE-KL and MVAE , which require exponential number of sub-networks and loss terms respectively . Secondly , we define the concept of importance in an unsupervised setting , and propose novel cross-correlation based loss terms to learn important regions in each modality ’ s representation space . The importances are modeled by separate unimodal networks referred to as importance networks . Hyper-parameter ρj for the j-th modality controls the integration of prior domain knowledge about the degree of importance in that modality . While not trained on any supervised labels , the learnt importances from IMA are analyzed quantitatively and found to correspond to the separation between digit vs. noise labels and emotion vs. neutral categories . 2 RELATED WORK . Following the great success of deep neural networks for representation learning , the research area of multimodal machine learning is gaining interest ( Baltrušaitis et al. , 2018 ) . Our proposed IMA model is relevant to two main research areas in this domain , Inter-modality Correlation Learning and Efficient Multimodal VAEs . The idea of learning acoustic embeddings for words has also been explored in Wang et al . ( 2018 ) and Jung et al . ( 2019 ) however we attempt to map words to their affective rather than phonetic representations . In this section , we describe each area and conclude with the similarities and differences between the IMA model and prior approaches . Inter-modality Correlation Learning : There have been several approaches which measure correlations between modalities/sources of data to understand how observed data in each modality can be explained by shared underlying concepts . The IBFA ( Inter-Battery Factor Analysis ) introduced by Tucker ( 1958 ) and its successor , the MBFA ( Multi-battery factor Analysis ) ( Browne , 1980 ) are among the earliest proposed techniques to study shared factors between score sets from batteries of tests . DeepCCA ( Deep Canonical Covariance Analysis ) proposed by Benton et al . ( 2017 ) learns a deep projection of each modality in a bimodal understanding scenario so that the projections are maximally correlated , effectively extending the classical CCA technique ( Knapp , 1978 ) to deep neural networks . Our proposed model extends these approaches to also detect important regions of each modality correlated with the shared latent factor . Efficient Multimodal VAEs : VAEs ( Variational Autoencoders ) have been applied to multimodal data for applications such as inference and bidirectional generation of modalities . This poses a major challenge of constructing encoders to model the latent posterior which are efficient in training/inference under any combination of input modalities . Recent work addresses this by focusing on factorized models for efficient inference . Vedantam et al . ( 2017 ) employs a product-of-experts decomposition with modality specific inference networks to train image generation models . Wu & Goodman ( 2018 ) propose MVAE ( Multimodal Variational Autoencoders ) , where the latent posterior is modeled with a parameter shared product of experts network . Shi et al . ( 2019 ) proposed a mixture-of-experts multimodal variational autoencoder ( MMVAE ) where the posterior is a mixture of experts instead . These approaches have been extended more recently , for example in multi-source neural variational inference ( Kurle et al. , 2019 ) where the multimodal posterior is constructed using learnt and integrated beliefs from multiple posteriors , each being informed by a different source . Sutter et al . ( 2020 ) introduce a novel Jensen-Shannon divergence based objective function which can be used to approximate both unimodal and joint multimodal posteriors . While existing approaches attempt to efficiently learn multimodal representations through posterior modeling , our proposed IMA model aligns modalities during autoencoder training for projection to a common space which facilitates inference even in absence of modalities . Only M encoders and O ( M ) loss terms are required by the IMA model for inference with M modalities . Prior work has also not focused sufficiently on unsupervised learning of modality importances ( through detection of subspaces maximally correlated with shared latent factors ) which we address in this paper . 3 MODEL DESCRIPTION . The IMA model consists of two main components : ( 1 ) the multimodal autoencoder and ( 2 ) the unimodal importance networks . In Figure 1 we have provided an overview diagram of the proposed model , including the loss functions utilized in training . Multimodal Autoencoder : Assume that the input training examples consist of multimodal data , where each multimodal sample is denoted as x = { x1 , x2 , ... xM } . There are M modalities and the training set consists of N multimodal samples . The input data xj in the j-th modality is passed through an encoder for that modality and its output is denoted as uj ( xj ) . The latent multimodal representation z could be modeled with different approaches for example , concatenated fusion with fully connected layers ( Suzuki et al. , 2016 ) . We wish to model the latent multimodal representation z ( x1 , x2 ... xM ) with a pooling weighted by distinct modality importances yj ( xj ) as given by : z ( x1 , x2 , ... xM ) = M∑ j=1 yj ( xj ) uj ( xj ) ∑ j yj ( xj ) = 1 ( 1 ) The multimodal representation z is passed through the decoder networks to obtain the reconstruction in the j-th modality as x̂j . L ( j ) rec is the reconstruction loss for the j-th modality . Lglob is a global regularization term which encourages z to be centered with zero mean . This term is for regularization of the multimodal representation z and expands to Lglob ( z ) = ‖z ( x1 , x2 , ... xM ) ‖2 . Multimodal alignment in IMA : For our proposed model , we have M sub-networks for each modality , where each j-th unimodal sub-network is trained using the autoencoder reconstruction losses and an additional alignment loss L ( j ) align . By forcing the multimodal representation z and each modality ’ s view uj ( xj ) to be similar , we also enforce each unimodal sub-network to learn its contribution to the latent factor z during autoencoder training . Per-sample , this requires only M sub-networks and M additional loss terms instead of a random subset of 2M losses for sub-sampled training as in the MVAE model ( Wu & Goodman , 2018 ) or 2M subnetworks as in the JMVAE-KL model ( Suzuki et al. , 2016 ) . Similar to MVAE , this model also can learn and infer in the absence of modalities . z can be expressed as a linear average of the unimodal representations present and incorporated into the overall loss . If the j-th modality is missing , we can set yj ( xj ) = 0 in Equation 1 . For the j-th modality , L ( j ) align is the SSE error between the multimodal representation z and the unimodal encoder outputs uj ( xj ) given by : L ( j ) align ( z , uj ) = ‖z ( x1 , x2 , ... xM ) − uj ( xj ) ‖ 2 When there are N training samples , during multimodal autoencoder learning the loss below is optimized : Lauto = λglob N∑ i=1 Lglob ( zi ) + [ N∑ i=1 M∑ j=1 λ ( j ) recL ( j ) rec ( xij , x̂ij ) ] + [ N∑ i=1 M∑ j=1 λ ( j ) alignL ( j ) align ( zi , uij ) ] ( 2 ) λalign , λrec and λglob are the associated hyper-parameter weights for the alignment , reconstruction and global multimodal regularization terms which appear in Equation ( 2 ) . Since it is not straightforward to tune such parameters for an unsupervised model , we start with equal weights for all losses and observe if that is sufficient to minimize all terms simultaneously during training . For the MNIST-TIDIGITS and IEMOCAP experiments , we have found that equally weighing them works , thus λalign = λrec = λglob = 1.0 . Importance Network Training : Section 1 explains that there are samples in observed data for each modality which do not correlate with the multimodal representation z . For the scope of this paper , we consider such samples as uncorrelated noise in the data . We have assumed the presence of a subspace Rj inside each j-th modality ’ s representation space so that if xj ∈ Rj , uj is minimally correlated with z . We seek to learn weights yij ∈ [ 0 , 1 ] for the i-th sample in the j−th modality , so that yij denotes the importance of each sample xij ( i.e . the degree to which xij does not belong to Rj ) . A unimodal neural network is trained to map from xij to yij for the j-th modality ; this is called the importance network . The multimodal autoencoder solely being trained to reconstruct all modality inputs would not have learnt yij corresponding to uncorrelated noise . This motivates the need for explicitly defining an importance-based loss . Cross-correlation Losses : To train the importance networks , we have made use of a loss function explicitly capturing cross-correlation between z and uj , weighted by yj . Minimizing this cost is equivalent to enforcing zero-correlation between uj and z based on mini-batch statistics during training , where size of a mini-batch is B samples . This cost can be derived from the definition of cross-correlation , where each i-th sample in the mini-batch is not weighted equally . Recent work has also applied other independence criterion , such as MMD ( Maximum Mean Discrepancies ) which act as auxiliary losses in a variational framework ( Louizos et al. , 2015 ) . Assuming the latent variables are Gaussians , independence and uncorrelated properties are equivalent , and we define the following alternative loss term based on the Frobenius norm of the cross-covariance between z and uj , where y′ij = 1− yij and w′ij = y′ij / B∑ i=1 y′ij L ( j ) corr = ∥∥∥∥∥ B∑ i=1 w′ij [ ( z− µz ) ( uj − µu ) ] T ∥∥∥∥∥ 2 F µz = B∑ i=1 w′ijzi µu = B∑ i=1 w′ijuij ( 3 ) Importance Priors : Learning to predict yij ≈ 1.0 ∀i , j trivially decreases Lcorr down to zero , and thus we need to regularize the importance network training , through an additional loss function which we refer to as L ( j ) local . This loss utilizes the hyper-parameter ρj ∈ [ 0 , 1 ] which serves as prior about how much of the j-th modality is corrupted by uncorrelated noise , and is defined as the KL-divergence DKL ( Bernoulli ( ȳj ) ||Bernoulli ( ρj ) ) where ȳj = ∑ i yij/B is the average value of yij as computed over a mini-batch of size B. Enforcing the loss Llocal at a sample level would have forced yij = ρj for all samples , which is addressed by defining the loss on a mini-batch instead . The importance network for each modality minimizes Limp , which is the weighted sum of the cross-covariance based loss and the regularization term , as defined below . λ ( j ) local and λ ( j ) corr are hyper-parameter weights for each loss term . Limp = N∑ i=1 M∑ j=1 λ ( j ) localL ( j ) local + N∑ i=1 M∑ j=1 λ ( j ) corrL ( j ) corr ( 4 ) It is important to note the difference between yij and y′ij = 1 − yij in terms of notation . yij is the importance of the i−th sample in the j−th modality and is required in the joint representation , whereas its complement y′ij is utilized in importance network training . The intuition is that a model minimizing Equation 3 would tend to learn high values of y′ij for maximally uncorrelated samples , which translates to low importance ( yij ) for these same samples .

Summary: The paper proposes the IMA model, a scalable model that learns modality importances and robust multimodal representations through a novel cross-covariance based loss function. The proposed model performs unimodal inference in absence of modalities and also addresses the problem of detecting important subspaces in each modality through weighted cross-covariance loss terms, which are minimized by unimodal importance networks. Results are shown that the IMA model is able to distinguish digits from uncorrelated noise, and word-level importances are learned that correspond to the separation between function and emotional words. The multimodal representations learned by IMA are also competitive with state-of-the-art baseline approaches on downstream tasks.