Maximum Likelihood Estimation for Multimodal Learning with Missing Modality
Authors: Fei Ma
Xiangxiang Xu
Shao-Lun Huang
Lin Zhang
Venue: ICLR 2022 Submitted (2022)
Keywords: multimodal learning
missing modality
maximum likelihood estimation

Abstract: Multimodal learning has achieved great successes in many scenarios. Compared with unimodal learning, it can effectively combine the information from different modalities to improve the performance of learning tasks. In reality, the multimodal data may have missing modalities due to various reasons, such as sensor failure and data transmission error. In previous works, the information of the modality-missing data has not been well exploited. To address this problem, we propose an efficient approach based on maximum likelihood estimation to incorporate the knowledge in the modality-missing data. Specifically, we design a likelihood function to characterize the conditional distributions of the modality-complete data and the modality-missing data, which is theoretically optimal. Moreover, we develop a generalized form of the softmax function to effectively implement maximum likelihood estimation in an end-to-end manner. Such training strategy guarantees the computability of our algorithm capably. Finally, we conduct a series of experiments on real-world multimodal datasets. Our results demonstrate the effectiveness of the proposed approach, even when 95% of the training data has missing modality.

Under review as a conference paper at ICLR 2022
MAXIMUM LIKELIHOOD ESTIMATION FOR MULTI- MODAL LEARNING WITH MISSING MODALITY
Anonymous authors Paper under double-blind review
ABSTRACT
Multimodal learning has achieved great successes in many scenarios. Compared with unimodal learning, can effectively combine the information from different modalities improve the performance of learning tasks. In reality, the multimodal data may have missing modalities due to various reasons, such as sensor failure and data transmission error. In previous works, the information of the modality- missing data has not been well exploited. To address this problem, we propose an efficient approach based on maximum likelihood estimation to incorporate the knowledge in the modality-mison function to characterize the conditional distributions of the modality-complete data and the modality-misssin data, which is theoretically optimal. Moreover, we develop generalized form the softmax function to effectively implement maximum likelihood estimation in end-to-end manner. Such training strategy guarantees the computability our algorithm capably. Finally, we conduct a series of experiments real-world multimodal datasets. Our results demonstrate the effectiveness of the proposed approach, even when 95% of the training data has missing modality.
data. Specifically, we design likelihood
INTRODUCTION
Multimodal learning is an important research area, which builds models to process and relate information between different modalities (Ngiam al., 2011; Srivastava Salakhutdinov [2014]
Baltrušaitis al. 2018). Compared with unimodal learning, multimodal learning can achieve better performance by properly utilizing the multimodal data. has been successfully used in many applications, such multimodal emotion recognition (Soleymani et al.) [011]] Mittal et al.
2020). multimedia event detection (Li et al. 2020), and visual duestion-answering
Yu et al.) (201)). With the emergence of big data, multimodal learning becomes more and more important combine the multimodal data from different sources.
A number of previous works (Tiiraiis al. 2017 Zhang et al. 2017 Elliott al. 2017: Kim et al.
2020; (Zhang et al.] 2020) have achieved great successes based on complete observations during the training process. However, in practice, the multimodal data may have missing modalities (Du et al., 2018; Ma et al., 2021alb b). This may be caused by various reasons. For instance, the sensor that collects the multimodal data is damaged or the network transmission fails. Examples of the multimodal data are shown in Figure
In the past years, different approaches have been proposed to deal with modality missing. A simple and typical way (Hastie et al. 2209) is to directly discard the data with missing modalities. Since the information contained in the modality miisiing data neglected, such method often has limited performance. In addition, researchers (Tran et al. 2017 Chen & Zhang 2020 Liu et al.] 2021 Ma al. 2021b) have proposed approaches to heuristically combine the information of the modality- missing data. However, most of these works lack theoretical explanations, and these empirical methods are often implemented using multiple training stages rather than an end-to-end manner, which lead the information of the modality- miisiig data not being well exploited.
To tackle above issues, we propose an efficient approach based on maximum likelihood estimation to effectively utilize the poodality-missing data. To be specific, we present likelihood function to characterize the conditional distributions of the modality complete data and the modality missing data, which is theoretically optimal. Furthermore, we adopt generalized form of the softmax

---

Under review as a conference paper at ICLR 2022
Visual:
Audio:
(a) Complete
(b) Visual Missing
(c) Audio Missing
Figure 1: Examples of the multimodal data: (a) complete observations, (b) observations which may have missing visual modality, and (c) observations which may have missing audio modality.
function efficiently implement maximum likelihood estimation algorithm. Such training strategy guarantees the computability of our framework an end-to-end scheme. this way, our approach can effectively leverage the information of the modality data during the training process, Finally, we perform several experiments on real-world multimodal datasets, including eNTERFACE 05 (Martin al. 2006) and RAVDESS (Livingstone & Russo 2018). The results show
the effectiveness of our approach in handling the problem of modality missing. To summarize, our contribution is three-fold:
We design a likelihood function to learn the conditional distributions of the modality- complete data and the modality-mise data, which is theoretically optimal.
We develop a generalized form the softmax function to implement our maximum likeli- hood estimation framework in end-to-end manner, which is more effective than previous works.
We conduct series of experiments on real-world multimodal datasets. The results validate the effectiveness of our approach, even when 95% of the training data has missing modality.
METHODOLOGY
Our goal is to deal with the problem of modality missing in multimodal learning based on maximum likelihood estimation. the following, we first show the problem formulation, and then describe the details of our framework.
2.1 PROBLEM FORMULATION
In this paper, we consider that the multimodal data has two modalities. Here, the random variables corresponding to these two modalities and their category labels are denoted as X, Y, and Z, re- spectively. In the training process, assume that there are two independently observed datasets:
modality-complete and modality-missing. We use Dxyz = ((*e We ze = =e E Z
(6)
(1,2, | represent the modality-complete dataset, where x:(") and ye represent the two modalities of the i-th sample of Dxyz respectively, (1) is their corresponding category label,
(i)
and the size of Dxyz is ne- We then use Dxz = (x 2m
(() 2m E Z = 21,2.... |2 } represent the modslity-missing dataset, where the size of Dxz is Tmn- In addition, adopt
[Dxyzlxy to represent {xxl" ye } [Dxyz]z. [Dx2lx. and [Dxzlz are expressed in the same way. The multimodal data of Dxyz and Dxz assumed to be i..d. generated from unknown underlying joint distribution. By utilizing the knowledge the modality-complete data and the modalltyymisssing data, hope framework can predict the category labels correctly.
(8)
2.2 MAXIMUM LIKELIHOOD ESTIMATION FOR MISSING MODALITY
In this section, first present how to design likelihood function learn the conditional distributions of the modality-come complete data and the poddality-missine data. Then, we show that by adopting a generalized form of the sofumax function, design training strategy to implement algorithm.
2

---

Under review as a conference paper at ICLR 2022
Visai Network
Class Label
"Anger"
Label Network
"Disgust"
Leg-likellhood Function
C
"Happiness""
Audio Network
"Sadness"
Our proposed framework for multimodal learning with missing modality. In the training process, we propose log-likelihood function L, shown Equation 2 learn the conditional distributions of the modality-complete data and the modality-miissngg data. By developing a general- ized form of the softmax function, we implement our maximum likelihood estimation algorithm in an end-to-end manner.
Figure 2:
2.2.1 LIKELIHOOD FUNCTION ANALYSES
Maximum likelihood estimation is statistical method of using the observed data to estimate the distribution by maximizing the likelihood function. The estimated distribution makes the observed data most likely (Myung) 2003). With this idea, we study the likelihood function on datasets Dxyz and Dxz. For the classification task, the conditional likelihood is commonly used. Inspired by this, we use model Q xxzz z to learn the underlying joint distribution of Dxyz and Dxz. The conditional likelihood can be represented as:
L P ([Dxyzlz, Dxzlz
(Pxaxxxxx
P ((Dxyzz (Dxy2lxy:Qxy2) ([Dx2]2 (xxll::Xxz2)
II
Qz(r((2/ry)
II Qz)x(=/=)
(xxy,))EDxvz
(xx=)eDxz
where the step a follows from the fact that datasets Dxyz z and Dxz are observed independently, and the step is due to that samples each dataset are i.d. Qzixy and Qz are conditional distributions of QxY z- In this way, show the likelihood function using the information of Dxyz and Dxz- Then, we use the negative log- ligeliiei as the loss function to train our deep learning model, i.e.,
llog
Σ
log Q2)xy(>(rr)
Σ
log Qzl (2/2)
(2)
(x.y.:)EDxYYZ
(xx
It is worth noting that in (Daniels) 1961 Lehmann 2004). maximum likelihood estimation is proved
to be an cumppooticllly-eficieeee strategy, which guarantees the theoretical optimality of our method deal with modality missing.
To optimize L, we use deep neural networks to extract the k-dimensional feature representations from the observation (x, 2), which are represented as fxx
f2(x), [x))² g(v) =
[9 (v). (3)) 9& (v)]*. and h(2) = [h, (2), h2(2).... hx))))² respectively. We then utilize these features to learn Qz xY and Qzix in L. Our framework is shown in Figure
In this way, show the log-likelihood function L. By characterizing the conditional distributions of the complet data and the modality-missing data, leverages the underlying structure information behind the multimodal data, which constitutes the theoretical basis of framework.
2.2.2 MAXIMUM LIKELIHOOD ESTIMATION IMPLEMENTATION
In fact, it is not easy to optimize the log- likelihood function L in Equation 22 by designing neural networks, which is mainly due to two reasons. Firstly, the representations the high-dimensional data and the procedure to model them are complicated. Secondly, since Q and Qzix x in are

---

Under review as a conference paper at ICLR 2022
9
g
g
(a) Addition
(b) Concutenation
(c) Outer product
Figure 3: Three forms & are studied: (a) addition [Wang et 2019 i.e., Φ(f.9) - f (b)
concatenation (Chandar al. 2016), i.e., g)) f ".'",, and (c) outer product (Zadeh et al. 2017 i... (ffgg - vec g). where vec represents the vectorization of outer product.
related, how to build models to learn their relationships is difficult. To address these two issues, develop a generalized form of the softmax function to describe QXYz as follows
(x (z) (v) exp( g)) Qxyz(E.B12) Ev' By (y/ (2 exp T (f (()))
(3)
where ((f.g) represents the function to fuse features and g. We study three forms of Φ to investigate its effect in framework, shown in Figure 3 Rx. Ry, and Rz represent the underlying marginal distributions the variables X, Y, and Z. respectively. Their use makes the denominator of Equation (3) expressed in the form of the mean over Rx, Ry, and Rz, which serves as the normalization to make x z a valid distribution and is helpful for our further derivation. In addition, the generalized softmax function we propose can regarded as generalization of softmax learning in (Xu et al. 2018) from unimodal learning to multimodal learning.
In this way, we show the distribution Qxyz by adopting generalized form of the softmax function, which has the following two benefits. Firstly, by depicting the representation of Q we can further derive Qz5x and @ It makes our approach a unified framework to combine the information of the modality-compp relationship between Q2 and Q In fact, the correlation between the high-dimensional data can be rather complex.
data and the modality data. Secondly, it avoids modeling the
Then, derive conditional distributions QzixY and Qzix from Equation
Qz(xy(=(ry) = R2(22) Eg Rz(2 (f))
(4)
and
Ey By (v') exp (f(o) @z(x(=|=)= R2(2) E. R2(2') Ey Ry (V') exp(x (f())
(5)
We observe that by introducing Rx, Ry, and Rz into Qxyz, 2, the derived Q and Qzx expressed in the form of the mean over Ry and Rz. In practice, we can use the empirical mean as an estimation. Correspondingly, by plugging Equations (4) and (5) into Equation (2) the detailed steps to compute our objective function L, as shown Algorithm It is worth pointing that when compute Q need to the information of the modality y. Since in the training process, the modality y of the dataset Dxz is missing, we utilize samples of the modality of the dataset Dxyz z compute Q
summarize
Finally, we utilize neural networks to extract features f. g. and from the modality-complett data and the poodality-missing data to optimize our log likelihood function It performs classification directly, which does not need to explicitly complement the modality modality-missing data before the classification task.
SStricly speaking, Rx and Hy are probability density functions, and Rz is probability mass function. The denominator of Equation 3 should be integrated over Rx and Ry. We use summation here for the simplicity of exposition.
4

---

Under review as a conference paper at ICLR 2022
Algorithm I Compute our objective function on mini-batch.
Input:
A codality batch { {(xt) Me .z(") ) where n, is the batch size.
(6)
A coddality-missingg batch { {am 2( Neural networks with A output units:
i=1' where ng is the batch size. f. g. and h.
122
Output:
The value of our objective L.
1: Compute empirical label distribution Rz:
R2(2) + Eis (xx("==)+ 1(2 1.2,
2: Compute QzixY:
(z)r( (ac .v6 Rz(20
(8)
ex (oo
i= 1,,,... ,....,,,
E (x(=')xx((")) (f J.ol Mo)))
3: Compute Qzix:
Qz) (0) (210 (2( _x'=1 Rz(2²) mi Σ exp(x gl Ma(e)) i 1, ,,. n2
- exp
4: Compute our empirical objective L:
-hs log (2e | :yc -E log Q21x (2m
(8)
Im
EXPERIMENTS
In this section, we first describe the multimodal datasets used in our experiment, then explain the experimental settings and baseline methods, and finally give the experimental results to show the effectiveness our approach.
3.1 DATASETS
We perform experiments on two public real-world multimodal datasets: eNTERFACE'05 (Martin itt al. 2006) and RAVDESS (Livingstone & Russo 2018). eNTERFACE'05 is an audio-visual emotion database in English. It contains 42 subjects eliciting the six basic emotions: anger, disgust, fear, happiness, sadness, and surprise. There are 213 videos for happiness, and 216 videos for cach of the remaining emotions. Following (Ma et al. 2020), we extract 30 segment samples from each video and then obtain a processed dataset with 38, 790 samples.
RAVDESS is multimodal database of emotional speech and song, which consists of 24 professional actors in a neutral North American accent. Here, we use the speech part, which includes calm, happy, sad, angry, fearful, surprise, and disgust expressions. Each recording is also in the video form. Similar to the eNTERFACE'O 05 dataset, we only consider six basic emotions, each of which has 5,760 segment samples.
3.2 EXPERIMENTAL SETTINGS
We perform experiments on the processed eNTERFACE'OS and RAVDESS datasets. Each segment of these two datasets has duration of 0.5 seconds. As shown in (Ma al. 2020p), consecutive frames within 0.5 seconds usually contain the same emotion in a similar way, which inspired us to choose the central frame of each segment the visual modality. This technique not only makes that the visual data contains enough emotional information, but also avoids the redundancy in multiple frames. Besides, the log Mel-spectrogram is extracted from each segment as the audio modality, which similar to the RGB image. We then feed these data into our framework obtain the classification result. ResNet-50 (He et al.) 2016) is used as the backbone visual network and audio network g extract features from visual and audio modalities, respectively. In addition, we transform the corresponding label into the one-hot form and then extract the label feature using label network with fully connected layer. f, g. and h are trained together.
On each processed dataset, we split all data into three parts: training set, validation set, and test set. Their proportions are 70%, 15%, and 15% In practice modality missing often occurs with high
5

---

Under review as a conference paper at ICLR 2022
Table 1: The classification performance with missing modality on the eNTERFACE'OS dataset.
Visual Missing
Audio Missing
Method
95%
90%
80%
95%
90%
80%
Addition
26.39
35.26
46.91
27.53
35.26
50.93
Lower Bound
Concatenation
27.11
36.39
46.49
27.84
33.71
46.29
Outer product
26.91
37.53
42.78
28.56
34.95
48.14
Addition
41.24
47.11
50.31
42.78
45.77
51.75
AE
Concatenation
43.92
46.39
50.00
42.16
47.42
50.72
Outer product
49.79
52.16
55.15
47.73
53.09
53.51
Addition
41.34
59.69
58.97
54.95
74.12
77.32
HGR MC
Concatenation
41.34
57.84
63.51
57.42
75.67
79.18
Outer product
49.90
59.69
64.64
55.46
76.29
77.94
Addition
58.66
67.84
69.07
76.49
78.35
80.41
ZP
Concatenation
60.21
67.11
68.76
76.70
78.25
80.93
Addition
66.29
71.65
72.37
79.38
80.31
81.24
Ours
Concatenation
64.74
70.82
72.27
79.79
80.21
81.24
Outer product
66.08
71.13
72.06
80.31
81.03
81.65
missing rate (Suo et al. 2019 Ma et al. 2021b). Here, in the training stage, we study three kinds of missing rates: 80%, 90%, and 95%. The case where the audio modality missing and the case where the visual modality is missing are investigated respectively. Following (Yu et al. 2020 Chen Zhang 2020: Du al., (022), we set modality missing arising during the training phase show that a large amount of unimodal data can assist the training of our multimodal learning framework. In the inference phase, we use Equation to predict the class label of the given test data.
Finally, we run each experiment five times and report average test accuracies to evaluate the perfor- mance of our approach and baseline methods. Adam optimizer (Kingma Ba 2015) is used to train neural networks with the learning rate of 0.0001. Both the size of modality size modality-missing batch are set to 90, The number of epochs set to 100. All experiments are implemented by Pytorch Passze et al. 2019) NVIDIA TITAN V GPU card.
batch and the
3.3 BASELINE METHODS
To show the effectiveness of our method, we compare our approach with the following methods which can also handle modality missing to some extent.
Discarding Modality-incomplete Data (Lower Bound): One simple strategy to handle modality missing is to directly discard the modality cedaalty-incompleee data, and then only use the modality-complete data for the classification task. This method does not use the information of the data with missing modalities. In our maximum likelihood estimation model, this equivalent to calculating Q2)Xy without calculating Qzix- Therefore, this method can also used as the ablation study of our approach.
Hirschfeld-GGeeeliin-eee 1959) (HGR MC): HGR MC is a statistical measure which calculates the dependence between different random variables. has been successfully used for multimodal learning (Ma et al. 2021a 2020 Wang et al. 2019; Xu & Huang 2020). Here, we use it further deal with modality missing. For the modality-complete data, we learn the maximal correlation between
Maximal Correlation (Hirschfeld 1935 Gebelein 1941 Rényi
and z. For the modality- data, we learn the maximal correlation between x and z.
Zero Padding (ZP): Padding the feature representation of the missing modality with zero is another widely used way to cope with incomplete modalities (Jo et al. 2019 Chen et al. 2020; Shen al. 2020). For ZP, consider forms of Φ to fuse features f and g addition and concatenation. The reason why the form of outer product is not studied here is that if the feature of one modality zero, the outer product of and the non-zero feature of another modality is also zero, which makes the modality-mistn
data useless.
Autoencoder (AE): An autoencoder a neural network framework used to learn the represen- tation from the training data. Some previous approaches apply autoencoders complement

---

Under review as a conference paper at ICLR 2022
Table 2: The classification performance with missing modality on the RAVDESS dataset.
Visual Missing
Audio Missing
Method
95%
90%
80%
95%
90%
80%
Addition
43.01
48.79
62.66
37.11
50.40
67.63
Lower Bound
Concatenation
41.27
49.94
62.66
35.61
49.48
67.63
Outer product
37.34
45.90
63.35
36.87
47.86
68.32
Addition
52.60
58.03
67.75
87.39
88.44
90.17
AE
Concatenation
53.29
58.03
67.75
87.39
88.21
90.29
Outer product
53.53
57.34
66.24
87.86
89.25
90.52
Addition
38.49
48.78
64.51
42,43
53.87
73.75
HGR MC
Concatenation
42.08
48.90
65.20
43.70
52.60
73.17
Outer product
37.34
46.59
62.43
43.01
51.33
70.98
Addition
44.74
52.72
67.40
83.81
86.24
90.17
ZP
Concatenation
43.70
52.02
65.89
82.31
86.47
90.63
Addition
56.64
60.34
70.06
89.72
90.52
91.33
Ours
Concatenation
58.84
61.27
69.83
89.25
89.94
91.91
Outer product
54.45
58.26
67.51
90.06
91.21
91.68
the missing modality (Liu et al. 2021 Jaques et al. 2017 Pereira et al. 2020 Shi et al. 2019). Following these works, for the modality-complett data, input of the autoencoder to reconstruct modality Then we use the trained autoencoder to predict the modality y impute the modality-missing data. Then we use the imputed data to perform the classification task. It is noted that the AE method has several stages while our method end-to-end.
modality æ as the
For fair comparison, we make that each method has the same network architecture and training strategy, and report the classification results of each method after the same number of repeated experiments.
3.4 EXPERIMENTAL RESULTS
We first conduct classification experiments the eNTERFACE'05 and RAVDESS datasets by comparing our framework with other methods. The experimental setting is shown in Section 3.2 We report the classification accuracy of each method in each setting. The results are shown in Table and Table 2
We have the following summarizations from Table[ant Table [ (1) The methods of AE, HGR MC, ZP, and ours can improve the classification accuracy compared to the Lower Bound method which only uses the modality-complee data. Our method achieves the highest classification performance among all methods under different settings. The higher the missing rate, the more obvious the gap between other methods and our method. These show that maximum likelihood estimation approach are more effective to tackle modality missing compared with other methods. (2) Different forms of will affect the classification performance. For example, for our approach, addition and outer product perform better than concatenation on the eNTERFACE'S 05 dataset. However, on the RAVDESS dataset, the concatenation form of achieves higher classification performance than the addition and outer product forms under some settings. This indicates that in different settings, the discrimination ability of the learned feature representations different. We need to design the appropriate form of Φ to fuse features of the multimodal data. (3) When the visual modality is missing, the classification accuracy is lower than that when the audio modality missing, indicating that the visual modality has more significant contribution to the classification performance, which consistent with previous works (Zhang et al. 2017 Ma et al. 2020).
In addition, we show the classification confusion matrices using the methods of AE, HGR MC, ZP, and ours when the missing rate of visual modality reaches 95% on the eNTERFACE 05 dataset, as shown in Figure/4, It can be seen that the classification accuracy of each emotion using AE HGR MC not high, which indicates that they can only deal with modality missing to a certain extent. The overall classification performance of 7P is lower than ours, but the classification accuracy of
7

---

Under review as a conference paper at ICLR 2022
01ge1
0nger
22.50
17.30
1.00
5.00
13.13
13.75
4.26
13.46
8.32
36.05
7.45
17.39
15.53
13.04
6.43
76.09
19.00
23.00
10.36
happthwox
3.75
19.38
8.13
$7.30
1.24
10.50
1.50
14.37
1.50
59.13
10.42
0.80
15.53
6.83
13.06
12.42
26.22
16.77
1.45
19.08
37:33
sagae
happines
Medrowk
sarpriee
saggr
dappot
lear
happltes
Melnosk
(a) AE
(b) HGR MC
71.69
5.46
1.00
7.27
1.21
3.48
3.00
1.82
8.03
2.42
dsguet
11:25
40.75
10.20
17.50
0.50
5.62
diggust
13.13
57.50
5.63
9.38
32
16.00
fear
15.53
6.21
15.53
8.07
fear
14
8.54
01
happhess
hoppinosa
Aadrwos
8.62
4.20
16.96
3.07
69.10
7.00
7.88
3.07
12.27
1.23
73.62
2:46
19.86
4,36
14.91
8.69
19.60
40
00
surprse
11.80
8.70
11.80
4.95
14
80
Bigan
bae
happinste
skinaso
onger
diggus
happaeee
Burprse
(c) ZP
(d) Ours
Figure 4: The confusion matrices of different methods on the eNTERFACE 05 dataset.
"happiness" is slightly higher than ours. This shows that different emotions have different clues for the classification task.
We then investigate the effect of the backbone in coping with modality missing. In the above experiments, we use ResNet-50 as the backbone of different methods extract feature representations. Here, we replace ResNet-50 with ResNet-34 (He et al. 2016) and VGG-16 (Simonyan & Zisserman 2015) respectively, and conduct experiments to compare the performance of different backbones when 95% of the training data has missing visual modality on the RAVDESS dataset, as shown in We can observe that compared with VGG-16 and ResNet-34, ResNet-50 achieves the highest performance. In addition, no matter what kind of backbone is based on, the classification accuracy using our method the highest, followed by using AE, ZP and HGR MC, and the lowest using Lower Bound, which shows that our approach take effect for different backbones.
Figure 5
RELATED WORKS
Multimodal learning has achieved great successes in many applications. An important topic this field is multimodal representations (Baltrušaitis et al.) 2018: Zhu et al.) (2020). which learn feature representations from the multimodal data by using the information of different modalities. How to learn good representations is investigated in (Ngiam et al. [011 Wu et al. 2014 Pan et 2016 Xu ct al., 2015). Another important topic is multimodal fusion (Atrey et al. 2010 Poria et al.) 2017) 7). which combines the information from different modalities to make predictions. Feature-based fusion one of the most common types of multimodal fusion. It concatenates the feature representations extracted from different modalities. This fusion approach is adopted by previous works (Tzirakis) (et al.) 2017] Zhang et al., 2017] Castellano et al.) 20088 [hhang et al. 2016). Modality missing is a key challenge in applying multimodal learning to the real world.
8

---

Under review as a conference paper at ICLR 2022
27
27
(a) VGG-16
(b) ResNet-34
(c) ResNet-50
Figure The performance comparison of different backbones on the RAVDESS dataset.
To cope with the problem of modality missing, a few methods have been proposed. For example, Ma et al. (202Ib) propose a Bayesian meta learning framework to perturb the latent feature space so that embeddings of single modality approximate embeddings of full modality. Tran et al. (2017)
propose a cascaded residual autoencoder for imputation with missing modalities, which is composed of set of stacked residual autoencoders that iteratively model the residuals. [Chen & Zhang (2020)
propose a heterogeneous graph-based multimodal fusion approach to enable multimodal fusion of incomplete data within heterogeneous graph structure. Liu al. (202T) propose an autoencoder framework to complement the missing data in the kernel space while taking into account the structural information of data and the inherent association between multiple views.
The above approaches can combine the information of the Our work is significantly different from them. The reason lies in the following two facts. Firstly, by exploiting the likelihood function to learn the conditional distributions of the modality-compler data and the modality-missin data, our method has theoretical guarantee, which is skipped by previous works. Secondly, the training process of our approach is in an end-to-end manner, while the training processes of most above methods are relatively cumbersome.
data to some extent.
CONCLUSION
Multimodal learning is hot topic in the academic and industry communities, of which key challenge is modality missing. In practice, the multimodal data may not be complete due to various reasons. Most previous works cannot effectively utilize the modality-missing data for the learning task. To address this problem, we propose an efficient approach to leverage the knowledge in the modality- missing data during the training stage. Specifically, we present framework based on maximum likelihood estimation to characterize the conditional distributions of the coodality-oomplte data and modality-missinn data, which has a theoretical guarantee. Furthermore, we develop a generalized form of the softmax function to effectively implement our maximum likelihood estimation framework in an end-to end-to-end way. We conduct experiments on the eNTERFACE" 05 dataset and the RAVDESS dataset for multimodal learning to demonstrate the effectiveness of our approach. In the future, can further extend our framework to other multimodal learning domains.
REPRODUCIBILITY STATEMENT
We provide our code in "supplement.zip". In this folder, "eNTERFACE, SAATTERFACEEpppprooeesssss "RAVDESS_ SecreSS_ppprroess.ppo py"" extract segment samples from the original videos of the eNTER- FACE'0S dataset and the RAVDESS dataset, respectively. "mle.py" shows the function compute our maximum likelihood estimation algorithm.
wwwpppcces..ppppppooopppppppeppp py" and
REFERENCES
Pradeep K Atrey, M Anwar Hossain, Abdulmotaleb EI Saddik, and Mohan S Kankanhalli. Multimodal fusion for multimedia analysis: survey. Multimedia systems, 16(6):345-379, 2010.

---

Under review as a conference paper at ICLR 2022
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philipee Morency. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2): 423 443, 2018.
Ginevra Castellano, Loic Kessous, and George Caridakis. Emotion recognition through multiple modalities: face, body gesture, speech. In Affect and emotion in kuman-compuler interaction, pp. 92-103. Springer, 2008.
Sarath Chandar, Mitesh M Khapra, Hugo Larochelle, and Balaraman Ravindran. Correlational neural networks. Neural computation, 28(2)2557-28, 2016.
Jiayi Chen and Aidong Zhang. Hgmf: Heterogeneous graph-based fusion for multimodal data with incompleteness. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp.
2020.
Zhengyang Chen, Shuai Wang, and Yanmin Qian. Multi-modality matters: A performance leap on voxceleb. Proc. Interspeech 2020, pp. 2252 2256, 2020.
HE Daniels. The asymptotic efficiency of maximum likelihood estimator. In Fourth Berkeley Symposium Mathematical Statistics and Probability, volume pp. 151-163. University of California Press Berkeley, 1961.
Changde Du, Changying Du, Hao Wang, Jinpeng Li, Wei-Long Zheng, Bao-Liang Lu, and Huiguang He. Semi-supervised deep generative modelling incomplete multi-modality emotional data. In Proceedings of the 26th ACM international conference Multimedia, pp. 108-116, 2018.
Changde Du, Changying Du, and Huiguang He. Multimodal deep generative adversarial models for scalable doubly semi-supervised learning. Information Fusion, 68: 1 18-130, 2021.
Desmond Elliott, Stella Frank. Loic Barrault, Fethi Bougares, and Lucia Specia. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference Machine Translation, pp. 215-233, 2017.
Hans Gebelein. Das statistische problem der korrelation als variations-und eigenwertproblem und sein zusammenhang mit der ausgleichsrechnung. 'AMMM-Jurral of Applied Mathematics and Mechanics/Zeitschrfff für Angewandte Mathematik und Mechanik, (1):3664 -399, 1941.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction. Springer Science & Business Media, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Hermann O Hirschfeld. A connection between correlation and contingency. In Mathematical Pro- ceedings of the Cambridge Philosophical Society, volume 31, pp. 520-524, Cambridge University Press, 1935.
Natasha Jaques, Sara Taylor, Akane Sano, and Rosalind Picard. Multimodal autoencoder: A deep learning approach to filling in missing sensor data and enabling better mood prediction. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII), pp- 202-208. IEEE, 2017.
Dae Ung Jo, ByeongJu Lee, Jongwon Choi, Haanju Yoo, and Jin Young Choi. Cross-modal variational auto-encoder with distributed latent spaces and associators. arXiv preprint wrXiv:1905.12800 2019.
Eun-Sol Kim, Woo Young Kang, Kyoung- Woon On, Yu-Jung Heo, and Byoung- Tak Zhang. Hyper- graph attention networks for multimodal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 14581-14590, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations (ICLR), 2015.
10

---

Under review as a conference paper at ICLR 2022
Erich Leo Lehmann. Elements of large-sample theory. Springer Science & Business Media, 2004.
Manling Li, Alireza Zareian, Qi Zeng, Spencer Whitehead, Di Lu, Heng Ji, and Shih-Fu Chang. Cross-media structured common space for multimedia event extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2557-2568, 2020.
Yanbei Liu, Lianxi Fan, Changqing Zhang, Tao Zhou, Zhitao Xiao, Lei Geng, and Dinggang Shen. Incomplete multi-modal representation learning for alzheimer's 's disease diagnosis. Medical Image Analysis, 69:101953, 2021.
Steven R Livingstone and Frank Russo. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 13(5):e0196391, 2018.
Fei Ma, Wei Zhang, Yang Li, Shao-Lun Lun Huang, and Lin Zhang. Learning better representations for audio-visual emotion recognition with common information. Applied Sciences, 10(20), 2020.
Fei Ma, Shao-Lun Huang, and Lin Zhang. An efficient approach for audio-visual emotion recognition with missing labels and missing modalities. In 2021 IEEE International Conference on Multimedia and Expo (ICME), 2021a.
Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng. Smil: Multimodal learning with severely missing modality. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 2302-2310, 2021b.
O. Martin, I. Kotsia, B. Macq, and Pitas. The enterface' 05 audio-visual emotion database. In 22nd International Conference Data Engineering Workshops (ICDEW'06) 066), pp. 8-8, 2006.
Trisha Mittal, Pooja Guhan, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha. Emoticon: Context-aware multimodal emotion recognition using frege's principle. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14234-14243, 2020.
In Jae Myung. Tutorial on maximum likelihood estimation. Journal of mathematical Psychology, 47 (1):90-100, 2003.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multi- modal deep learning. In ICML, 2011.
Yingwei Pan, Tao Mei, Ting Yao. Houqiang Li, and Yong Rui. Jointly modeling embedding and translation to bridge video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4594-4602. 2016.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurlPS, pp. 8026-8037, 2019.
Ricardo Cardoso Pereira, Miriam Scoane Santos, Pedro Pereira Rodrigues, and Pedro Henriques Abreu. Reviewing autoencoders for missing data imputation: Technical trends, applications and outcomes. Journal of Artificial Intelligence Research, 69: 19.1555-1285, 2020.
Soujanya Poria, Erik Cambria, Rajiv Bajpai, and Amir Hussain. A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion, 37:98-125, 2017.
Alfréd Rényi. On measures of dependence. Acta Mathematica Academiae Scientiarum Hungarica, 10(3-4):44-4551 1959.
Guangyao Shen, Xin Wang, Xuguang Duan, Hongzhi Li, and Wenwu Zhu. Memor: A dataset for multimodal emotion reasoning in videos. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 493-502, 2020.
Yuge Shi, Brooks Paige, Philip Torr, al. Variational mixture-of of-eoperts autoencoders for multi- modal deep generative models. Advances in Neural Information Processing Systems, 32:15718- 15729, 2019.
11

---

Under review as a conference paper at ICLR 2022
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. International Conference on Learning Representations (ICLR), 2015.
Mohammad Soleymani, Maja Pantic, and Thierry Pun. Multimodal emotion recognition in response to videos. IEEE transactions on affective computing, ((2):211-223, 2011.
Nitish Srivastava and Ruslan Salakhutdinov. Multimodal learning with deep boltzmann machines. Journal of Machine Learning Research, 15(84):2949-299) 2980, 2014.
Qiuling Suo, Weida Zhong, Fenglong Ma, Ye Yuan, Jing Gao, and Aidong Zhang. Metric learning on healthcare data with incomplete modalities. In IJCAI, pp. 3534-3540, 2019.
Luan Tran, Xiaoming Liu, Jiayu Zhou, and Rong Jin. Missing modalities imputation via cascaded residual autoencoder. In 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 4971 4971-4980, 2017.
Panagiotis Tzirakis, George Trigeorgis, Mihalis A Nicolaou, Björn W Schuller, and Stefanos Zafeiriou. End-to-end multimodal emotion recognition using deep neural networks. IEEE Journal of Selected Topics in Signal Processing, 1 1188) 2017.
Lichen Wang, Jiaxiang Wu, Shao-Lun Huang, Lizhong Zheng, Xiangxiang Xu, Lin Zhang, and Junzhou Huang. An efficient approach to informative feature extraction from multimodal data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 5281-5288 2019.
Zuxuan Wu, Yu-Gang Jiang, Jun Wang, Jian Pu, and Xiangyang Xue. Exploring inter-feature and inter-class relationships with deep neural networks for video classification. In Proceedings of the 22nd ACM international conference Multimedia, pp. 167-176, 2014.
Ran Xu, Caiming Xiong, Wei Chen, and Jason Corso. Jointly modeling deep video and compositional text to bridge vision and language in unified framework. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
Xiangxiang Xu and Shao-Lun Huang. Maximal correlation regression. IEEE Access, 8:26591-26601, 2020.
Xiangxiang Xu, Shao-Lun Huang, Lizhong Zheng, and Lin Zhang. The geometric structure of generalized softmax learning. In 2018 IEEE Information Theory Workshop (ITW), pp. 1-5, 2018.
Guan Yu, Quefeng Li, Dinggang Shen, and Yufeng Liu. Optimal sparse linear prediction for block- missing multi-modality data without imputation. Journal of the American Statistical Association, 115(531): 106-:14:111119 2020.
Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6281-6290 2019.
Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency, Tensor fusion network for multimodal sentiment analysis. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1103-1114, 2017.
Huaizheng Zhang, Linsen Dong, Guanyu Gao, Han Hu, Yonggang Wen, and Kyle Guan. Deepqoe: A multimodal learning framework for video quality of experience (qoe) prediction. JEEE Transactions on Multimedia, 22 22(12):3210-322 2020.
Shiqing Zhang, Shiliang Zhang, Tiejun Huang, and Wen Gao. Multimodal deep convolutional neural network for audio-visual emotion recognition. In Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval, pp. 281-284, 2016
Shiqing Zhang, Shiliang Zhang, Tiejun Huang, Wen Gao, and Qi Tian. Learning affective features with a hybrid deep model for audio-visual emotion recognition. IEEE Transactions Circuits and Systems for Video Technology, 280100300303343 2017.
Wenwu Zhu, Xin Wang, and Wen Gao. Multimedia intelligence: When multimedia meets artificial 2020.
intelligence, IEFE Transactions Multimedia,
12


Reviews:
[{"review_id": "tx6-JjQWvhk", "reviewer": "Reviewer_cK5q", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution of the paper is the proposal of the generalized softmax function, to model the joint distribution of all modalities and the label. The generalized softmax function consists of the product of the marginal distributions of the modalities and the label, as if they were independent, subsequently compensating for this (the dependence among the modalities and the label) via an exponential function, enhanced with feature extraction models. This joint distribution leads to computationally efficient conditional distributions.\n\nHowever, there are a few concerns about the approaches and the evaluations in this paper:\n\n* The most significant concern is about the baseline comparison.  The authors set these baselines as instances of specifically defined simpler models (or their own model) in order to highlight specific manner in which those models deal with the missing modality. However, there are many prior works focused on solving the exact missing modality problem (see below). The authors should, thus, compare against those baselines instead of deriving their own baseline model instances.  [1] Multimodal Generative Models for Scalable Weakly-Supervised Learning (Wu and Goodman); [2] Private-Shared Disentangled Multimodal VAE for Learning of Latent\nRepresentations (Lee and Pavlovic); [3] MHVAE: a Human-Inspired Deep Hierarchical Generative Model for Multimodal Representation Learning (Vasco, Melo, and Paiva).\n\n* In the experiments (Tab.1 and Tab.2), the visual missing rate and the audio missing rates are likely those in the training set. It is not clear what are the missing rates are for the testing set, if any.  The authors should clarify this.\n\n* In Fig.4 (c) and (d), the classification accuracy of the “happiness” emotion by ZP is higher than that by the proposed method. This single value may be caused by several factors, hidden or accidental. Thus, it may not be sound to claim that “the proposed method is more efficient to exploit the information in most categories”. To support that claim, more investigation is needed and the authors should present it.\n", "summary_of_the_paper": "The authors propose a probabilistic framework to improve the classification accuracy in instances when there exists missing data in the multi-modality datasets (where one of the modalities is the predictive label; however, this label is not assumed missing). To this end, they propose a generalized softmax function as the joint distribution of all modalities and the label, from which conditional distributions are derived, for computing the maximum likelihood estimate (MLE). Experimental results on eNTERFACE and RAVDESS datasets demonstrate improvements in classification accuracy over baselines. In addition, the authors investigate the influence of the influence of the backbone models and the fusion functions.\n", "summary_of_the_review": "I find the claims in the paper largely unsubstantiated due to the lack of comparison to baselines as mentioned in the Main Review above. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "t9uJgchxzy6", "reviewer": "Reviewer_APG5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n-\tMultimodal learning has achieved great success for many applications and having missing modality is an important challenge to be tackled. This paper presents a simple, end-to-end method, in a way novel and contributing to the field as being based on maximum likelihood, which is not presented by prior art.\n\nWeakness:\n-\tThe biggest limitation of this work is considering only the multimodal data having two modalities as well as not even discussing how the proposed method behaves in case of having more modalities.\n-\tAlthough the paper presents a task-free method, the experimental analysis were limited to two datasets, both are addressing the same task: emotion recognition. As also mentioned in the introduction there are several other tasks that the proposed method could have been tested on. Indeed, the related work (such as Ma et al. SMIL: \"Multimodal Learning with Severely Missing Modality\".) was tested on several different tasks. I suggest authors to either change the paper including the title, abstract and related work and target categorical emotion recognition or comprehensively extend the experimental analysis such that the proposed method would be tested and validated on several other tasks. \n-\tIt is also important to mention that the proposed method was tested only for categorical emotion recognition, while emotion datasets are typically multi-labeled (as humans cannot elicit only one emotion at a time) and also include continuous values therefore, regression task might be targeted too\n-\tI also found the used datasets limited in terms of their size. In case authors would like to keep the emotion recognition task as the testbed, I suggest them using a much larger dataset called: CMU-MOSEI, also having other modalities than video and audio. Indeed some related work was tested on CMU-MOSEI and/or CMU-MOSI such as Ma et al. SMIL: \"Multimodal Learning with Severely Missing Modality\".\n-\tAnother limitation regarding the experimental analysis performed is that: as modality only visual and audio data were used. Testing on combinations of several other modalities: text, depth data, data of mocap, accelerometer, gyro-meter would improve the validity of the proposed method.\n-\t“In addition, the generalized softmax function we propose….” Generalized softmax function might be misleading, it more sounds like the used softmax has tolerance to the diversity of samples belonging to different classes or somehow a domain adaption is being applied. But these are not the cases.\n-\tIt is unclear why authors think that the multimodal softmax is a contribution. Eq. 3 and following equations look like standard softmax was written for multimodal data instead of first fusing the data and representing the fused data as a single feature vector. On the other hand, the fusion of the data is performed through standard strategies: addition, concatenation and multiplication. I expect authors to clarify the contribution in this respect.\n-\tThere are also lack of information regarding how the data is being processed. In detail: \na)\tWhat does “we take central frame” as visual modality mean? Do you take a bunch of frames and use only the central frame? If so what is the motivation behind this? What is the window size? In fact, it is more frequent to apply spatio-temporal processing, for example, processing motion and appearance in facial images for emotion recognition. Thus, I do not understand the rationale behind discarding the temporal information.\nb)\tAnother issue is reading the audio data; it is not clear what the audio data chunk selected to calculate the log Mel-spectrogram.\nc)\t“On each processed dataset, we split all data into three parts: training set, validation set, and test set. Their proportions are 70%, 15%, and 15%.” Are you randomly picking these splits and applying sort of a k-fold cross validation or these splits are obtained only once and fixed? Do you guarantee that you use exactly the same split for all baseline methods, this is a matter because the used datasets are relatively small? I am aware that prior art on emotion recognition uses 5 or 10 fold cross validation for the same datasets, and I am not sure why authors have selected a different data splitting strategy.\n-\t  The proposed method was compared with some relatively simpler baselines such as zero padding, but the comparative study should include the SOTA methods, e.g., Ma et al (2021b), Tran et al. (2017), Chen & Zhang (2020), Liu et al. (2021), Suo et al., (2019). Given this lack of comparison, I believe that the claim of authors “……which lead to the information of the modality-missing data not being well exploited to” was not justified as well.\n-\tI believe authors should include a better discussion why they tackle with the missing data only in training but never take into account that there could be missing modality in testing as well. I think in a practical scenario it is more possible to train a model with a full set of modalities, while during test some of the modalities are either completely or only for some test samples missing.\n-\tTables should include the results of unimodal data processing to allow reader to understand which modality perform better than other when used alone, and include the results of processing complete data (i.e., no missing modality) as the upper bound.\n-\t“When the visual modality is missing, the classification accuracy is lower than that when the audio modality is missing, indicating that the visual modality has a more significant contribution to the classification performance, which is consistent with previous works (Zhang et al., 2017; Ma et al., 2020).” I believe the citations in this sentence is a bit irrelevant. In detail, the authors are not using neither the same feature sets nor the same datasets with the cited works.\n\n\n", "summary_of_the_paper": "This paper deals with multimodal learning with missing modality in training. Specifically, the proposed method is based on maximum likelihood estimation to obtain the conditional distributions of the so-called \"the modality complete data\" and \"the modality-missing data\" in which a multimodal softmax function is defined to implement this framework in an end-to-end manner.", "summary_of_the_review": "The experimental analysis is limited in terms of several aspects: a) applications: tested only for emotion recognition, b) fixed type of modalities: only audio and video, c) no comparisons with the SOTA ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "365Nmpe6ZG-", "reviewer": "Reviewer_wkEE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. In Introduction , the authors states that \"Compared with unimodal learning, multimodal learning can effectively utilize the multimodal data to achieve better performance.\", actually in some cases multimodal data must be utilized properly to make multimodal learning more effective  than unimodal learning. For e.g., researchers have found that the best unimodal model can outperform its multimodal counterpart in this paper: \"W. Wang, et al. What Makes Training Multi-modal Classification Networks Hard?\" The authors should try to make the statements more accurate.\n\n2. The author have mentioned in Page 4 that \"In the following, we will show that we use empirical distribution to implement these underlying marginal distribution in our algorithm.\", but the reviewer could not find any descriptions in the following paragraphs.\n\n3. In experiments on eNTERFACES’05, the condition with 100% missing rate should be considered, which could be helpful to demonstrate whether the left 5% data in 95%-missing case is indeed used for the task or just because the other complete modality of data.\n\n4. The authors mentioned several different methods dealing with missing modality in related works, but no experiments to compare the performances between the proposed methods and the mentioned framework. At the same time, the comparative methods in this submission are less persuasive.", "summary_of_the_paper": "This submission proposed a maximum likelihood estimation framework combined with a generalized softmax function to resolve multimodal emotion recognition with missing modality. Two emotion recognition datasets are used in experiments to make comparison with several baseline methods. The results suggest that the proposed approach outperforms these compared methods. Moreover, according to the authors, the end-to-end nature of this framework makes it more efficient than previous works.", "summary_of_the_review": "There is a mismatch between the title and the content, given that the proposed methods are only verified in one application scenario, i.e., emotion recognition. However, the multimodal learning is such a big topic, including but not limited to emotion recognition, action recognition, etc. The authors may consider to extend the methods in order to match the title or change the title to a specific area.\n\nBesides, the experiments the authors have conducted is far from extensive, and the comparative methods are not sufficent to support the conclusion.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three experts reviewed this paper and all recommended rejection. The rebuttal did not change the reviewers' recommendations. The reviewers was not excited by the proposed probabilistic framework and raised many concerns regarding the comparison with baselines and competing methods, limited size of datasets, and limited scope of one dataset for one task. Considering the reviewers' concerns, we regret that the paper cannot be recommended for acceptance at this time.  The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.
Decision: Reject