Year: 2021

Abstract:
Multimodal learning has achieved great successes in many scenarios. Compared with unimodal learning, it can effectively combine the information from different modalities to improve the performance of learning tasks. In reality, the multimodal data may have missing modalities due to various reasons, such as sensor failure and data transmission error. In previous works, the information of the modalitymissing data has not been well exploited. To address this problem, we propose an efficient approach based on maximum likelihood estimation to incorporate the knowledge in the modality-missing data. Specifically, we design a likelihood function to characterize the conditional distributions of the modality-complete data and the modality-missing data, which is theoretically optimal. Moreover, we develop a generalized form of the softmax function to effectively implement maximum likelihood estimation in an end-to-end manner. Such training strategy guarantees the computability of our algorithm capably. Finally, we conduct a series of experiments on real-world multimodal datasets. Our results demonstrate the effectiveness of the proposed approach, even when 95% of the training data has missing modality.

heading: 1 INTRODUCTION
text: Multimodal learning is an important research area, which builds models to process and relate information between different modalities (Ngiam et al., 2011; Srivastava & Salakhutdinov, 2014; Baltrušaitis et al., 2018). Compared with unimodal learning, multimodal learning can achieve better performance by properly utilizing the multimodal data. It has been successfully used in many applications, such as multimodal emotion recognition (Soleymani et al., 2011; Mittal et al., 2020), multimedia event detection (Li et al., 2020), and visual question-answering (Yu et al., 2019). With the emergence of big data, multimodal learning becomes more and more important to combine the multimodal data from different sources.
A number of previous works (Tzirakis et al., 2017; Zhang et al., 2017; Elliott et al., 2017; Kim et al., 2020; Zhang et al., 2020) have achieved great successes based on complete observations during the training process. However, in practice, the multimodal data may have missing modalities (Du et al., 2018; Ma et al., 2021a;b). This may be caused by various reasons. For instance, the sensor that collects the multimodal data is damaged or the network transmission fails. Examples of the multimodal data are shown in Figure 1.
In the past years, different approaches have been proposed to deal with modality missing. A simple and typical way (Hastie et al., 2009) is to directly discard the data with missing modalities. Since the information contained in the modality-missing data is neglected, such method often has limited performance. In addition, researchers (Tran et al., 2017; Chen & Zhang, 2020; Liu et al., 2021; Ma et al., 2021b) have proposed approaches to heuristically combine the information of the modalitymissing data. However, most of these works lack theoretical explanations, and these empirical methods are often implemented using multiple training stages rather than an end-to-end manner, which lead to the information of the modality-missing data not being well exploited.
To tackle above issues, we propose an efficient approach based on maximum likelihood estimation to effectively utilize the modality-missing data. To be specific, we present a likelihood function to characterize the conditional distributions of the modality-complete data and the modality-missing data, which is theoretically optimal. Furthermore, we adopt a generalized form of the softmax
function to efficiently implement our maximum likelihood estimation algorithm. Such training strategy guarantees the computability of our framework in an end-to-end scheme. In this way, our approach can effectively leverage the information of the modality-missing data during the training process, Finally, we perform several experiments on real-world multimodal datasets, including eNTERFACE’05 (Martin et al., 2006) and RAVDESS (Livingstone & Russo, 2018). The results show the effectiveness of our approach in handling the problem of modality missing. To summarize, our contribution is three-fold:
• We design a likelihood function to learn the conditional distributions of the modalitycomplete data and the modality-missing data, which is theoretically optimal.
• We develop a generalized form of the softmax function to implement our maximum likelihood estimation framework in an end-to-end manner, which is more effective than previous works.
• We conduct a series of experiments on real-world multimodal datasets. The results validate the effectiveness of our approach, even when 95% of the training data has missing modality.

heading: 2 METHODOLOGY
text: Our goal is to deal with the problem of modality missing in multimodal learning based on maximum likelihood estimation. In the following, we first show the problem formulation, and then describe the details of our framework.

heading: 2.1 PROBLEM FORMULATION
text: In this paper, we consider that the multimodal data has two modalities. Here, the random variables corresponding to these two modalities and their category labels are denoted as X , Y , and Z, respectively. In the training process, we assume that there are two independently observed datasets: modality-complete and modality-missing. We use DXY Z = { (x (i) c , y (i) c , z (i) c ) | z(i)c ∈ Z = {1, 2, · · · , |Z|} }nc i=1 to represent the modality-complete dataset, where x(i)c and y (i) c represent the two modalities of the i-th sample of DXY Z respectively, z (i) c is their corresponding category label, and the size of DXY Z is nc. We then use DXZ = { (x (i) m , z (i) m ) | z(i)m ∈ Z = {1, 2, · · · , |Z|} }nm i=1 to represent the modality-missing dataset, where the size of DXZ is nm. In addition, we adopt [DXY Z ]XY to represent { (x (i) c , y (i) c ) }nc i=1
. [DXY Z ]Z , [DXZ ]X , and [DXZ ]Z are expressed in the same way. The multimodal data of DXY Z and DXZ are assumed to be i.i.d. generated from an unknown underlying joint distribution. By utilizing the knowledge of the modality-complete data and the modality-missing data, we hope our framework can predict the category labels correctly.

heading: 2.2 MAXIMUM LIKELIHOOD ESTIMATION FOR MISSING MODALITY
text: In this section, we first present how to design a likelihood function to learn the conditional distributions of the modality-complete data and the modality-missing data. Then, we show that by adopting a generalized form of the softmax function, we design a training strategy to implement our algorithm.

heading: 2.2.1 LIKELIHOOD FUNCTION ANALYSES
text: Maximum likelihood estimation is a statistical method of using the observed data to estimate the distribution by maximizing the likelihood function. The estimated distribution makes the observed data most likely (Myung, 2003). With this idea, we study the likelihood function on datasets DXY Z and DXZ . For the classification task, the conditional likelihood is commonly used. Inspired by this, we use a model QXY Z to learn the underlying joint distribution of DXY Z and DXZ . The conditional likelihood can be represented as:
` , P ([DXY Z ]Z , [DXZ ]Z | [DXY Z ]XY , [DXZ ]X ;QXY Z) a = P ([DXY Z ]Z | [DXY Z ]XY ;QXY Z) · P ([DXZ ]Z | [DXZ ]X ;QXY Z) b = ∏
(x,y,z)∈DXY Z
QZ|XY (z|xy) · ∏
(x,z)∈DXZ
QZ|X(z|x) (1)
where the step a follows from the fact that datasets DXY Z and DXZ are observed independently, and the step b is due to that samples in each dataset are i.i.d. QZ|XY and QZ|X are conditional distributions of QXY Z . In this way, we show the likelihood function using the information of DXY Z and DXZ . Then, we use the negative log-likelihood as the loss function to train our deep learning model, i.e.,
L , − log ` = − ∑
(x,y,z)∈DXY Z
logQZ|XY (z|xy)− ∑
(x,z)∈DXZ
logQZ|X(z|x) (2)
It is worth noting that in (Daniels, 1961; Lehmann, 2004), maximum likelihood estimation is proved to be an asymptotically-efficient strategy, which guarantees the theoretical optimality of our method to deal with modality missing.
To optimize L, we use deep neural networks to extract the k-dimensional feature representations from the observation (x, y, z), which are represented as f(x) = [f1(x), f2(x), · · · , fk(x)]T, g(y) = [g1(y), g2(y), · · · , gk(y)]T, and h(z) = [h1(z), h2(z), · · · , hk(z)]T, respectively. We then utilize these features to learn QZ|XY and QZ|X in L. Our framework is shown in Figure 2. In this way, we show the log-likelihood function L. By characterizing the conditional distributions of the modality-complete data and the modality-missing data, it leverages the underlying structure information behind the multimodal data, which constitutes the theoretical basis of our framework.

heading: 2.2.2 MAXIMUM LIKELIHOOD ESTIMATION IMPLEMENTATION
text: In fact, it is not easy to optimize the log-likelihood function L in Equation (2) by designing neural networks, which is mainly due to two reasons. Firstly, the representations of the high-dimensional data and the procedure to model them are complicated. Secondly, since QZ|XY and QZ|X in L are
related, how to build models to learn their relationships is difficult. To address these two issues, we develop a generalized form of the softmax function to describe QXY Z as follows 1:
QXY Z(x, y, z) = RX(x)RY (y)RZ(z) exp(φ T(f(x), g(y))h(z))∑ x′,y′,z′ RX(x ′)RY (y′)RZ(z′) exp(φT(f(x′), g(y′))h(z′)) (3)
where φ(f , g) represents the function to fuse features f and g. We study three forms of φ to investigate its effect in our framework, as shown in Figure 3. RX , RY , and RZ represent the underlying marginal distributions of the variables X , Y , and Z, respectively. Their use makes the denominator of Equation (3) expressed in the form of the mean over RX , RY , and RZ , which serves as the normalization to make QXY Z a valid distribution and is helpful for our further derivation. In addition, the generalized softmax function we propose can be regarded as a generalization of softmax learning in (Xu et al., 2018) from unimodal learning to multimodal learning.
In this way, we show the distribution QXY Z by adopting a generalized form of the softmax function, which has the following two benefits. Firstly, by depicting the representation of QXY Z , we can further derive QZ|XY and QZ|X . It makes our approach a unified framework to combine the information of the modality-complete data and the modality-missing data. Secondly, it avoids modeling the relationship between QZ|XY and QZ|X . In fact, the correlation between the high-dimensional data can be rather complex.
Then, we derive conditional distributions QZ|XY and QZ|X from Equation (3):
QZ|XY (z|xy) = RZ(z) exp(φT(f(x), g(y))h(z))∑
z′ RZ(z ′) exp(φT(f(x), g(y))h(z′))
(4)
and
QZ|X(z|x) = RZ(z) ∑ y′ RY (y ′) exp(φT(f(x), g(y′))h(z))∑
z′ RZ(z ′) ∑ y′ RY (y ′) exp(φT(f(x), g(y′))h(z′))
(5)
We can observe that by introducing RX , RY , and RZ into QXY Z , the derived QZ|XY and QZ|X are expressed in the form of the mean over RY and RZ . In practice, we can use the empirical mean as an estimation. Correspondingly, by plugging Equations (4) and (5) into Equation (2), we can summarize the detailed steps to compute our objective function L, as shown in Algorithm 1. It is worth pointing out that when we compute QZ|X , we need to use the information of the modality y. Since in the training process, the modality y of the dataset DXZ is missing, we utilize samples of the modality y of the dataset DXY Z to compute QZ|X .
Finally, we utilize neural networks to extract features f , g, and h from the modality-complete data and the modality-missing data to optimize our log-likelihood functionL. It performs classification directly, which does not need to explicitly complement the modality-missing data before the classification task.
1Strictly speaking, RX and RY are probability density functions, and RZ is a probability mass function. The denominator of Equation (3) should be integrated over RX and RY . We use summation here for the simplicity of exposition.
Algorithm 1 Compute our objective function on a mini-batch. Input:
A modality-complete batch { (x (i) c , y (i) c , z (i) c ) }n1 i=1 , where n1 is the batch size.
A modality-missing batch { (x (i) m , z (i) m ) }n2 i=1
, where n2 is the batch size. Neural networks with k output units: f , g, and h.
Output: The value of our objective L.
1: Compute empirical label distribution R̂Z : R̂Z(z)← ∑n1 i=1 1(z (i) c =z)+ ∑n2 i=1 1(z (i) m =z)
n1+n2 , z = 1, 2, · · · , |Z|
2: Compute QZ|XY :
QZ|XY (z (i) c |x(i)c , y(i)c )← R̂Z(z(i)c ) exp(φ T(f(x(i)c ),g(y (i) c ))h(z))∑|Z|
z′=1 RZ(z ′) exp(φT(f(x (i) c ),g(y (i) c ))h(z′)) , i = 1, · · · , n1 3: Compute QZ|X :
QZ|X(z (i) m |x(i)m ) ← R̂Z(z(i)m )
1 n1 ∑n1 j=1 exp(φ T(f(x(i)m ),g(y (j) c ))h(z
(i) m ))∑|Z|
z′=1 RZ(z ′) 1n1 ∑n1 j=1 exp(φ T(f(x (i) m ),g(y (j) c ))h(z′)) , i = 1, · · · , n2
4: Compute our empirical objective L: −∑n1i=1 logQZ|XY (z (i) c |x(i)c , y(i)c )− ∑n2 i=1 logQZ|X(z (i) m |x(i)m )

heading: 3 EXPERIMENTS
text: In this section, we first describe the real-world multimodal datasets used in our experiment, then explain the experimental settings and baseline methods, and finally give the experimental results to show the effectiveness of our approach.

heading: 3.1 DATASETS
text: We perform experiments on two public real-world multimodal datasets: eNTERFACE’05 (Martin et al., 2006) and RAVDESS (Livingstone & Russo, 2018). eNTERFACE’05 is an audio-visual emotion database in English. It contains 42 subjects eliciting the six basic emotions: anger, disgust, fear, happiness, sadness, and surprise. There are 213 videos for happiness, and 216 videos for each of the remaining emotions. Following (Ma et al., 2020), we extract 30 segment samples from each video and then obtain a processed dataset with 38,790 samples.
RAVDESS is a multimodal database of emotional speech and song, which consists of 24 professional actors in a neutral North American accent. Here, we use the speech part, which includes calm, happy, sad, angry, fearful, surprise, and disgust expressions. Each recording is also in the video form. Similar to the eNTERFACE’05 dataset, we only consider six basic emotions, each of which has 5,760 segment samples.

heading: 3.2 EXPERIMENTAL SETTINGS
text: We perform experiments on the processed eNTERFACE’05 and RAVDESS datasets. Each segment of these two datasets has a duration of 0.5 seconds. As shown in ((Ma et al., 2020)), consecutive frames within 0.5 seconds usually contain the same emotion in a similar way, which inspired us to choose the central frame of each segment as the visual modality. This technique not only makes that the visual data contains enough emotional information, but also avoids the redundancy in multiple frames. Besides, the log Mel-spectrogram is extracted from each segment as the audio modality, which is similar to the RGB image. We then feed these data into our framework to obtain the classification result. ResNet-50 (He et al., 2016) is used as the backbone of visual network f and audio network g to extract features from visual and audio modalities, respectively. In addition, we transform the corresponding label into the one-hot form and then extract the label feature using label network h with a fully connected layer. f , g, and h are trained together.
On each processed dataset, we split all data into three parts: training set, validation set, and test set. Their proportions are 70%, 15%, and 15%. In practice, modality missing often occurs with a high
missing rate (Suo et al., 2019; Ma et al., 2021b). Here, in the training stage, we study three kinds of missing rates: 80%, 90%, and 95%. The case where the audio modality is missing and the case where the visual modality is missing are investigated respectively. Following (Yu et al., 2020; Chen & Zhang, 2020; Du et al., 2021), we set modality missing arising during the training phase to show that a large amount of unimodal data can assist the training of our multimodal learning framework. In the inference phase, we use Equation (4) to predict the class label of the given test data.
Finally, we run each experiment five times and report average test accuracies to evaluate the performance of our approach and baseline methods. Adam optimizer (Kingma & Ba, 2015) is used to train neural networks with the learning rate of 0.0001. Both the size of modality-complete batch and the size of modality-missing batch are set to 90. The number of epochs is set to 100. All experiments are implemented by Pytorch (Paszke et al., 2019) on a NVIDIA TITAN V GPU card.

heading: 3.3 BASELINE METHODS
text: To show the effectiveness of our method, we compare our approach with the following methods which can also handle modality missing to some extent.
• Discarding Modality-incomplete Data (Lower Bound): One simple strategy to handle modality missing is to directly discard the modality-incomplete data, and then only use the modality-complete data for the classification task. This method does not use the information of the data with missing modalities. In our maximum likelihood estimation model, this is equivalent to calculating QZ|XY without calculating QZ|X . Therefore, this method can also be used as the ablation study of our approach.
• Hirschfeld-Gebelein-Renyi Maximal Correlation (Hirschfeld, 1935; Gebelein, 1941; Rényi, 1959) (HGR MC): HGR MC is a statistical measure which calculates the dependence between different random variables. It has been successfully used for multimodal learning (Ma et al., 2021a; 2020; Wang et al., 2019; Xu & Huang, 2020). Here, we use it further to deal with modality missing. For the modality-complete data, we learn the maximal correlation between x, y, and z. For the modality-missing data, we learn the maximal correlation between x and z.
• Zero Padding (ZP): Padding the feature representation of the missing modality with zero is another widely used way to cope with incomplete modalities (Jo et al., 2019; Chen et al., 2020; Shen et al., 2020). For ZP, we consider two forms of φ to fuse features f and g: addition and concatenation. The reason why the form of outer product is not studied here is that if the feature of one modality is zero, the outer product of it and the non-zero feature of another modality is also zero, which makes the modality-missing data useless.
• Autoencoder (AE): An autoencoder is a neural network framework used to learn the representation from the training data. Some previous approaches apply autoencoders to complement
For a fair comparison, we make that each method has the same network architecture and training strategy, and report the classification results of each method after the same number of repeated experiments.

heading: 3.4 EXPERIMENTAL RESULTS
text: We first conduct classification experiments on the eNTERFACE’05 and RAVDESS datasets by comparing our framework with other methods. The experimental setting is shown in Section 3.2. We report the classification accuracy of each method in each setting. The results are shown in Table 1 and Table 2.
We have the following summarizations from Table 1 and Table 2: (1) The methods of AE, HGR MC, ZP, and ours can improve the classification accuracy compared to the Lower Bound method which only uses the modality-complete data. Our method achieves the highest classification performance among all methods under different settings. The higher the missing rate, the more obvious the gap between other methods and our method. These show that our maximum likelihood estimation approach are more effective to tackle modality missing compared with other methods. (2) Different forms of φ will affect the classification performance. For example, for our approach, addition and outer product perform better than concatenation on the eNTERFACE’05 dataset. However, on the RAVDESS dataset, the concatenation form of φ achieves higher classification performance than the addition and outer product forms under some settings. This indicates that in different settings, the discrimination ability of the learned feature representations is different. We need to design the appropriate form of φ to fuse features of the multimodal data. (3) When the visual modality is missing, the classification accuracy is lower than that when the audio modality is missing, indicating that the visual modality has a more significant contribution to the classification performance, which is consistent with previous works (Zhang et al., 2017; Ma et al., 2020).
In addition, we show the classification confusion matrices using the methods of AE, HGR MC, ZP, and ours when the missing rate of visual modality reaches 95% on the eNTERFACE’05 dataset, as shown in Figure 4. It can be seen that the classification accuracy of each emotion using AE or HGR MC is not high, which indicates that they can only deal with modality missing to a certain extent. The overall classification performance of ZP is lower than ours, but the classification accuracy of
“happiness” is slightly higher than ours. This shows that different emotions have different clues for the classification task.
We then investigate the effect of the backbone in coping with modality missing. In the above experiments, we use ResNet-50 as the backbone of different methods to extract feature representations. Here, we replace ResNet-50 with ResNet-34 (He et al., 2016) and VGG-16 (Simonyan & Zisserman, 2015) respectively, and conduct experiments to compare the performance of different backbones when 95% of the training data has missing visual modality on the RAVDESS dataset, as shown in Figure 5. We can observe that compared with VGG-16 and ResNet-34, ResNet-50 achieves the highest performance. In addition, no matter what kind of backbone is based on, the classification accuracy using our method is the highest, followed by using AE, ZP and HGR MC, and the lowest using Lower Bound, which shows that our approach can take effect for different backbones.

heading: 4 RELATED WORKS
text: Multimodal learning has achieved great successes in many applications. An important topic in this field is multimodal representations (Baltrušaitis et al., 2018; Zhu et al., 2020), which learn feature representations from the multimodal data by using the information of different modalities. How to learn good representations is investigated in (Ngiam et al., 2011; Wu et al., 2014; Pan et al., 2016; Xu et al., 2015). Another important topic is multimodal fusion (Atrey et al., 2010; Poria et al., 2017), which combines the information from different modalities to make predictions. Feature-based fusion is one of the most common types of multimodal fusion. It concatenates the feature representations extracted from different modalities. This fusion approach is adopted by previous works (Tzirakis et al., 2017; Zhang et al., 2017; Castellano et al., 2008; Zhang et al., 2016). Modality missing is a key challenge in applying multimodal learning to the real world.
To cope with the problem of modality missing, a few methods have been proposed. For example, Ma et al. (2021b) propose a Bayesian meta learning framework to perturb the latent feature space so that embeddings of single modality can approximate embeddings of full modality. Tran et al. (2017) propose a cascaded residual autoencoder for imputation with missing modalities, which is composed of a set of stacked residual autoencoders that iteratively model the residuals. Chen & Zhang (2020) propose a heterogeneous graph-based multimodal fusion approach to enable multimodal fusion of incomplete data within a heterogeneous graph structure. Liu et al. (2021) propose an autoencoder framework to complement the missing data in the kernel space while taking into account the structural information of data and the inherent association between multiple views.
The above approaches can combine the information of the modality-missing data to some extent. Our work is significantly different from them. The reason lies in the following two facts. Firstly, by exploiting the likelihood function to learn the conditional distributions of the modality-complete data and the modality-missing data, our method has a theoretical guarantee, which is skipped by previous works. Secondly, the training process of our approach is in an end-to-end manner, while the training processes of most above methods are relatively cumbersome.

heading: 5 CONCLUSION
text: Multimodal learning is a hot topic in the academic and industry communities, of which a key challenge is modality missing. In practice, the multimodal data may not be complete due to various reasons. Most previous works cannot effectively utilize the modality-missing data for the learning task. To address this problem, we propose an efficient approach to leverage the knowledge in the modalitymissing data during the training stage. Specifically, we present a framework based on maximum likelihood estimation to characterize the conditional distributions of the modality-complete data and the modality-missing data, which has a theoretical guarantee. Furthermore, we develop a generalized form of the softmax function to effectively implement our maximum likelihood estimation framework in an end-to-end way. We conduct experiments on the eNTERFACE’05 dataset and the RAVDESS dataset for multimodal learning to demonstrate the effectiveness of our approach. In the future, we can further extend our framework to other multimodal learning domains.
REPRODUCIBILITY STATEMENT
We provide our code in “supplement.zip”. In this folder, “eNTERFACE_preprocess.py” and “RAVDESS_preprocess.py” extract segment samples from the original videos of the eNTERFACE’05 dataset and the RAVDESS dataset, respectively. “mle.py” shows the function to compute our maximum likelihood estimation algorithm.

Review summaries:
The authors propose a probabilistic framework to improve the classification accuracy in instances when there exists missing data in the multi-modality datasets (where one of the modalities is the predictive label; however, this label is not assumed missing). To this end, they propose a generalized softmax function as the joint distribution of all modalities and the label, from which conditional distributions are derived, for computing the maximum likelihood estimate (MLE). Experimental results on eNTERFACE and RAVDESS datasets demonstrate improvements in classification accuracy over baselines. In addition, the authors investigate the influence of the influence of the backbone models and the fusion functions. 

This submission proposed a maximum likelihood estimation framework combined with a generalized softmax function to resolve multimodal emotion recognition with missing modality. Two emotion recognition datasets are used in experiments to make comparison with several baseline methods. The results suggest that the proposed approach outperforms these compared methods. Moreover, according to the authors, the end-to-end nature of this framework makes it more efficient than previous works.