Maximum Likelihood Estimation for Multimodal Learning with Missing Modality

1 INTRODUCTION . Multimodal learning is an important research area , which builds models to process and relate information between different modalities ( Ngiam et al. , 2011 ; Srivastava & Salakhutdinov , 2014 ; Baltrušaitis et al. , 2018 ) . Compared with unimodal learning , multimodal learning can achieve better performance by properly utilizing the multimodal data . It has been successfully used in many applications , such as multimodal emotion recognition ( Soleymani et al. , 2011 ; Mittal et al. , 2020 ) , multimedia event detection ( Li et al. , 2020 ) , and visual question-answering ( Yu et al. , 2019 ) . With the emergence of big data , multimodal learning becomes more and more important to combine the multimodal data from different sources . A number of previous works ( Tzirakis et al. , 2017 ; Zhang et al. , 2017 ; Elliott et al. , 2017 ; Kim et al. , 2020 ; Zhang et al. , 2020 ) have achieved great successes based on complete observations during the training process . However , in practice , the multimodal data may have missing modalities ( Du et al. , 2018 ; Ma et al. , 2021a ; b ) . This may be caused by various reasons . For instance , the sensor that collects the multimodal data is damaged or the network transmission fails . Examples of the multimodal data are shown in Figure 1 . In the past years , different approaches have been proposed to deal with modality missing . A simple and typical way ( Hastie et al. , 2009 ) is to directly discard the data with missing modalities . Since the information contained in the modality-missing data is neglected , such method often has limited performance . In addition , researchers ( Tran et al. , 2017 ; Chen & Zhang , 2020 ; Liu et al. , 2021 ; Ma et al. , 2021b ) have proposed approaches to heuristically combine the information of the modalitymissing data . However , most of these works lack theoretical explanations , and these empirical methods are often implemented using multiple training stages rather than an end-to-end manner , which lead to the information of the modality-missing data not being well exploited . To tackle above issues , we propose an efficient approach based on maximum likelihood estimation to effectively utilize the modality-missing data . To be specific , we present a likelihood function to characterize the conditional distributions of the modality-complete data and the modality-missing data , which is theoretically optimal . Furthermore , we adopt a generalized form of the softmax function to efficiently implement our maximum likelihood estimation algorithm . Such training strategy guarantees the computability of our framework in an end-to-end scheme . In this way , our approach can effectively leverage the information of the modality-missing data during the training process , Finally , we perform several experiments on real-world multimodal datasets , including eNTERFACE ’ 05 ( Martin et al. , 2006 ) and RAVDESS ( Livingstone & Russo , 2018 ) . The results show the effectiveness of our approach in handling the problem of modality missing . To summarize , our contribution is three-fold : • We design a likelihood function to learn the conditional distributions of the modalitycomplete data and the modality-missing data , which is theoretically optimal . • We develop a generalized form of the softmax function to implement our maximum likelihood estimation framework in an end-to-end manner , which is more effective than previous works . • We conduct a series of experiments on real-world multimodal datasets . The results validate the effectiveness of our approach , even when 95 % of the training data has missing modality . 2 METHODOLOGY . Our goal is to deal with the problem of modality missing in multimodal learning based on maximum likelihood estimation . In the following , we first show the problem formulation , and then describe the details of our framework . 2.1 PROBLEM FORMULATION . In this paper , we consider that the multimodal data has two modalities . Here , the random variables corresponding to these two modalities and their category labels are denoted as X , Y , and Z , respectively . In the training process , we assume that there are two independently observed datasets : modality-complete and modality-missing . We use DXY Z = { ( x ( i ) c , y ( i ) c , z ( i ) c ) | z ( i ) c ∈ Z = { 1 , 2 , · · · , |Z| } } nc i=1 to represent the modality-complete dataset , where x ( i ) c and y ( i ) c represent the two modalities of the i-th sample of DXY Z respectively , z ( i ) c is their corresponding category label , and the size of DXY Z is nc . We then use DXZ = { ( x ( i ) m , z ( i ) m ) | z ( i ) m ∈ Z = { 1 , 2 , · · · , |Z| } } nm i=1 to represent the modality-missing dataset , where the size of DXZ is nm . In addition , we adopt [ DXY Z ] XY to represent { ( x ( i ) c , y ( i ) c ) } nc i=1 . [ DXY Z ] Z , [ DXZ ] X , and [ DXZ ] Z are expressed in the same way . The multimodal data of DXY Z and DXZ are assumed to be i.i.d . generated from an unknown underlying joint distribution . By utilizing the knowledge of the modality-complete data and the modality-missing data , we hope our framework can predict the category labels correctly . 2.2 MAXIMUM LIKELIHOOD ESTIMATION FOR MISSING MODALITY . In this section , we first present how to design a likelihood function to learn the conditional distributions of the modality-complete data and the modality-missing data . Then , we show that by adopting a generalized form of the softmax function , we design a training strategy to implement our algorithm . 2.2.1 LIKELIHOOD FUNCTION ANALYSES . Maximum likelihood estimation is a statistical method of using the observed data to estimate the distribution by maximizing the likelihood function . The estimated distribution makes the observed data most likely ( Myung , 2003 ) . With this idea , we study the likelihood function on datasets DXY Z and DXZ . For the classification task , the conditional likelihood is commonly used . Inspired by this , we use a model QXY Z to learn the underlying joint distribution of DXY Z and DXZ . The conditional likelihood can be represented as : ` , P ( [ DXY Z ] Z , [ DXZ ] Z | [ DXY Z ] XY , [ DXZ ] X ; QXY Z ) a = P ( [ DXY Z ] Z | [ DXY Z ] XY ; QXY Z ) · P ( [ DXZ ] Z | [ DXZ ] X ; QXY Z ) b = ∏ ( x , y , z ) ∈DXY Z QZ|XY ( z|xy ) · ∏ ( x , z ) ∈DXZ QZ|X ( z|x ) ( 1 ) where the step a follows from the fact that datasets DXY Z and DXZ are observed independently , and the step b is due to that samples in each dataset are i.i.d . QZ|XY and QZ|X are conditional distributions of QXY Z . In this way , we show the likelihood function using the information of DXY Z and DXZ . Then , we use the negative log-likelihood as the loss function to train our deep learning model , i.e. , L , − log ` = − ∑ ( x , y , z ) ∈DXY Z logQZ|XY ( z|xy ) − ∑ ( x , z ) ∈DXZ logQZ|X ( z|x ) ( 2 ) It is worth noting that in ( Daniels , 1961 ; Lehmann , 2004 ) , maximum likelihood estimation is proved to be an asymptotically-efficient strategy , which guarantees the theoretical optimality of our method to deal with modality missing . To optimize L , we use deep neural networks to extract the k-dimensional feature representations from the observation ( x , y , z ) , which are represented as f ( x ) = [ f1 ( x ) , f2 ( x ) , · · · , fk ( x ) ] T , g ( y ) = [ g1 ( y ) , g2 ( y ) , · · · , gk ( y ) ] T , and h ( z ) = [ h1 ( z ) , h2 ( z ) , · · · , hk ( z ) ] T , respectively . We then utilize these features to learn QZ|XY and QZ|X in L. Our framework is shown in Figure 2 . In this way , we show the log-likelihood function L. By characterizing the conditional distributions of the modality-complete data and the modality-missing data , it leverages the underlying structure information behind the multimodal data , which constitutes the theoretical basis of our framework . 2.2.2 MAXIMUM LIKELIHOOD ESTIMATION IMPLEMENTATION . In fact , it is not easy to optimize the log-likelihood function L in Equation ( 2 ) by designing neural networks , which is mainly due to two reasons . Firstly , the representations of the high-dimensional data and the procedure to model them are complicated . Secondly , since QZ|XY and QZ|X in L are related , how to build models to learn their relationships is difficult . To address these two issues , we develop a generalized form of the softmax function to describe QXY Z as follows 1 : QXY Z ( x , y , z ) = RX ( x ) RY ( y ) RZ ( z ) exp ( φ T ( f ( x ) , g ( y ) ) h ( z ) ) ∑ x′ , y′ , z′ RX ( x ′ ) RY ( y′ ) RZ ( z′ ) exp ( φT ( f ( x′ ) , g ( y′ ) ) h ( z′ ) ) ( 3 ) where φ ( f , g ) represents the function to fuse features f and g. We study three forms of φ to investigate its effect in our framework , as shown in Figure 3 . RX , RY , and RZ represent the underlying marginal distributions of the variables X , Y , and Z , respectively . Their use makes the denominator of Equation ( 3 ) expressed in the form of the mean over RX , RY , and RZ , which serves as the normalization to make QXY Z a valid distribution and is helpful for our further derivation . In addition , the generalized softmax function we propose can be regarded as a generalization of softmax learning in ( Xu et al. , 2018 ) from unimodal learning to multimodal learning . In this way , we show the distribution QXY Z by adopting a generalized form of the softmax function , which has the following two benefits . Firstly , by depicting the representation of QXY Z , we can further derive QZ|XY and QZ|X . It makes our approach a unified framework to combine the information of the modality-complete data and the modality-missing data . Secondly , it avoids modeling the relationship between QZ|XY and QZ|X . In fact , the correlation between the high-dimensional data can be rather complex . Then , we derive conditional distributions QZ|XY and QZ|X from Equation ( 3 ) : QZ|XY ( z|xy ) = RZ ( z ) exp ( φT ( f ( x ) , g ( y ) ) h ( z ) ) ∑ z′ RZ ( z ′ ) exp ( φT ( f ( x ) , g ( y ) ) h ( z′ ) ) ( 4 ) and QZ|X ( z|x ) = RZ ( z ) ∑ y′ RY ( y ′ ) exp ( φT ( f ( x ) , g ( y′ ) ) h ( z ) ) ∑ z′ RZ ( z ′ ) ∑ y′ RY ( y ′ ) exp ( φT ( f ( x ) , g ( y′ ) ) h ( z′ ) ) ( 5 ) We can observe that by introducing RX , RY , and RZ into QXY Z , the derived QZ|XY and QZ|X are expressed in the form of the mean over RY and RZ . In practice , we can use the empirical mean as an estimation . Correspondingly , by plugging Equations ( 4 ) and ( 5 ) into Equation ( 2 ) , we can summarize the detailed steps to compute our objective function L , as shown in Algorithm 1 . It is worth pointing out that when we compute QZ|X , we need to use the information of the modality y . Since in the training process , the modality y of the dataset DXZ is missing , we utilize samples of the modality y of the dataset DXY Z to compute QZ|X . Finally , we utilize neural networks to extract features f , g , and h from the modality-complete data and the modality-missing data to optimize our log-likelihood functionL . It performs classification directly , which does not need to explicitly complement the modality-missing data before the classification task . 1Strictly speaking , RX and RY are probability density functions , and RZ is a probability mass function . The denominator of Equation ( 3 ) should be integrated over RX and RY . We use summation here for the simplicity of exposition . Algorithm 1 Compute our objective function on a mini-batch . Input : A modality-complete batch { ( x ( i ) c , y ( i ) c , z ( i ) c ) } n1 i=1 , where n1 is the batch size . A modality-missing batch { ( x ( i ) m , z ( i ) m ) } n2 i=1 , where n2 is the batch size . Neural networks with k output units : f , g , and h. Output : The value of our objective L. 1 : Compute empirical label distribution R̂Z : R̂Z ( z ) ← ∑n1 i=1 1 ( z ( i ) c =z ) + ∑n2 i=1 1 ( z ( i ) m =z ) n1+n2 , z = 1 , 2 , · · · , |Z| 2 : Compute QZ|XY : QZ|XY ( z ( i ) c |x ( i ) c , y ( i ) c ) ← R̂Z ( z ( i ) c ) exp ( φ T ( f ( x ( i ) c ) , g ( y ( i ) c ) ) h ( z ) ) ∑|Z| z′=1 RZ ( z ′ ) exp ( φT ( f ( x ( i ) c ) , g ( y ( i ) c ) ) h ( z′ ) ) , i = 1 , · · · , n1 3 : Compute QZ|X : QZ|X ( z ( i ) m |x ( i ) m ) ← R̂Z ( z ( i ) m ) 1 n1 ∑n1 j=1 exp ( φ T ( f ( x ( i ) m ) , g ( y ( j ) c ) ) h ( z ( i ) m ) ) ∑|Z| z′=1 RZ ( z ′ ) 1n1 ∑n1 j=1 exp ( φ T ( f ( x ( i ) m ) , g ( y ( j ) c ) ) h ( z′ ) ) , i = 1 , · · · , n2 4 : Compute our empirical objective L : −∑n1i=1 logQZ|XY ( z ( i ) c |x ( i ) c , y ( i ) c ) − ∑n2 i=1 logQZ|X ( z ( i ) m |x ( i ) m )

Summary: This submission proposed a maximum likelihood estimation framework combined with a generalized softmax function to resolve multimodal emotion recognition with missing modality. Two emotion recognition datasets are used in experiments to make comparison with several baseline methods. The results suggest that the proposed approach outperforms these compared methods. Moreover, according to the authors, the end-to-end nature of this framework makes it more efficient than previous works.