Incomplete Multimodality-Diffused Emotion Recognition
Authors: Yuanzhi Wang
Yong Li
Zhen Cui
Venue: NeurIPS 2023 poster (2023)
Keywords: Multimodal emotion recognition
Incomplete multimodalities

Abstract: Human multimodal emotion recognition (MER) aims to perceive and understand human emotions via various heterogeneous modalities, such as language, vision, and acoustic. Compared with unimodality, the complementary information in the multimodalities facilitates robust emotion understanding. Nevertheless,  in real-world scenarios, the missing modalities hinder multimodal understanding and result in degraded MER performance. In this paper, we propose an Incomplete Multimodality-Diffused emotion recognition (IMDer) method to mitigate the challenge of MER under incomplete multimodalities. To recover the missing modalities, IMDer exploits the score-based diffusion model that maps the input Gaussian noise into the desired distribution space of the missing modalities and recovers missing data abided by their original distributions. Specially, to reduce semantic ambiguity between the missing and the recovered modalities, the available modalities are embedded as the condition to guide and refine the diffusion-based recovering process. In contrast to previous work,  the diffusion-based modality recovery mechanism in IMDer allows to simultaneously reach both distribution consistency and semantic disambiguation. Feature visualization of the recovered modalities illustrates the consistent modality-specific distribution and semantic alignment. Besides, quantitative experimental results verify that IMDer obtains state-of-the-art MER accuracy under various missing modality patterns.

Incomplete Duffi Emotion Recognition
Yuanzhi Wang, Yong Li, Zhen Cui PCA Lab, Key Lab of Intelligent Perception and Systems for High- SithnDiinnsi Information of Ministry of Education, School Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China. Cyuanzhiwang, yong.
zhen. cui]@njust edu.cn
Abstract
Human multimodal emotion recognition (MER) aims to perceive and understand human emotions various heterogeneous modalities, such as language, vision, and acoustic. Compared with unimodality, the complementary information in the multimodalities facilitates robust emotion understanding. Nevertheless, real-world scenarios, the missing modalities hinder multimodal understanding and result in degraded MER performance. this paper, we propose an Incom- plete Multimodality-De Diffussd emotion recognition (IMDer) method to mitigate the challenge MER under incomplete multimodalities. To recover the missing modalities, IMDer exploits the score-based diffusion model that maps the input Gaussian noise into the desired distribution space of the missing modalities and recovers missing data abided by their original distributions. Specially, to reduce semantic ambiguity between the missing and the recovered modalities, the available modalities embedded the condition to guide and refine the diffusion- based recovering process. In contrast to previous work, the diffusion-basee modality recovery mechanism in IMDer allows to simultaneously reach both distribution consistency and semantic disambiguation. Feature visualization of the recovered modalities illustrates the consistent modality-specioe distribution and semantic alignment. Besides, quantitative experimental results verify that IMDer obtains state-of-the-ard MER accuracy under various missing modality patterns. Codes are released at https:://gtt giighub..... com com/mdes mdswyz/ IIMDe..
Introduction
Benefiting from the intrinsic heterogeneity of multimodal data, various modalities are exploited for multimodal emotion recognition (MER) to understand human behaviors and intents from collaborative perspective [27, 13, 36]. Recently, MER has become one of the most active research topics of affective computing with many applications, such as healthcare [3, 18] and robotics [11, 12].
Robust MER relies learning and combining representations from diverse modalities [27]. In previous studies, Zadeh et al. [31] designed a Tensor Fusion Network that takes as input paired modalities encode the bimodal representation, which are further fused to generate the trimodal representation. Tsai et al. [27] proposed a multimodal transformer to learn the potential adaptation and correlations between modalities. Subsequently, various advanced approaches [17, 15, 13] have explored different variants of multimodal transformers to construct robust MER frameworks.
However, in real-world scenarios, not all modalities are consistently available, e.g., language data may be missing due to speech recognition errors; video data may be inaccessible due to privacy
Corresponding authors: Yong Li, Zhen Cui
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Available
Modality
Posttive
Postivv
Encoder
Decoder
Negstive
Condition
X
Classifeer
Negative
Missinn
Modality
Denoising
Xm
p(Xm|xa))
Recovered
Common Modality Recovery based MER Method
Proposed
IMDer
Modality
Figure 1: (a) shows the convenient modality recovery baaser MER method. Typically, well well-crafted incodrrddccoer architecture is used to recover the missing modalities. (b) illustrates our proposed Incomplete Multimodllty-Diffusss emotion recognition (IMDer) method. IMDer exploits the vanilla available modalities an the condition to guide and refine the distribution space of the recovered modality, thus achieves distribution consistency and semantic disambiguation.
and security concerns [14]. Ultimately, these incomplete multimodal data significantly hinder the performance of MER. For MER under incomplete multimodalities, one straightforward way is to recover the missing modalities from the available ones. As shown in Fig. (a), convenient modality recovery methods [20, 35, 14, 30] aim to recover the missing modalities via a well-designed encoder- decoder framework and build mapping between the vanilla available and the missing modalities. Among them, Zhao et al. [35] combined autoencoder with cycle consistency learning for modality recovery. Lian er al. [14] designed graph completion network that utilized graph neural networks to reconstruct missing parts. However, these previous methods fail to explicitly consider the modality- specific distributions that are highly correlated with each modality's 's intrinsic discriminability. e.g., an image gives the visual appearance of happy face via thousands of pixels, whilst the corresponding text describes this emotion with a sentence using discrete words.
In this paper, we aim to challenge MER under incomplete multimodalities by proposing an Incomplete dultimodality-Diffuuses
emotion recognition (IMDer) method, as shown in Fig. 1 (b). To recover the missing modalities, IMDer exploits the prevalent score-based diffusion model [24] that maps the input random noise into the distribution space of the missing modalities. In particular, the score-based diffusion model captures the distribution of the missing modalities by perturbing data with the stochastic differential equation (SDE). With sufficient data and model capacity, we are capable of exploiting the well-trained score model to recover the ceabbbutioncoosistteet modality by solving reverse-time SDE (i.e., denoising process) starting from the prior noise distribution.
To reduce the semantic ambiguity between the missing and the corresponding recovered modalities, the vanilla available modalities as semantic conditions to guide and refine the recovering process. The mere information embedded in the available modalities facilitates IMDer simultaneously reaching both distribution consistency and semantic disambiguation. Finally, the recovered modalities together with available modalities could be jointly fed into a multimodal fusion and prediction network for MER. In summary, the contributions of this work can be concluded as:
To address the challenge of MER under incomplete multimodalities, we propose the In- complete Multimodality-Dffunee
emotion recognition (IMDer) method. IMDer maps input random noise to the distribution space of missing modalities and recovers missing data in accordance with their original distributions.
To minimize the semantic ambiguity between the missing and recovered modalities, utilize the available modalities as prior conditions to guide and refine the recovering process. This ensures that the recovered modality is both distribution and cemantic-consistent.
We perform extensive experiments publicly available MER datasets and achieve superior or comparable results across different missing modality patterns. The feature visualization of the recovered modalities demonstrates consistent distribution and semantic alignment.
2 Background
2.1 Incomplete Multimodal Learning
Incomplete multimodal learning is an essential research topic in multimodal machine learning, it can deal with the inevitable modality-missing environments real-world scenarios. One effective approach is to find dow-dimensions subspace shared by all modalities, in which different modalities

---

have maximum correlation with each other [6, 2. 29]. However, this approach may ignore the complementarity between heterogeneous modalities, leading to suboptimal results for downstream tasks. Another more effective approach is to explicitly recover the missing modalities based on available ones. Representative methods include zero-based recovery [19], average- preedge-based recovery [34], and deep learning-bai baseg recovery [20]. Of these, the zero and the average-based recovery methods still cause a considerable gap between the recovered and the original data because they do not utilize any supervision information. In contrast, deep learning-bass -baee methods can better estimate missing modalities by leveraging their powerful feature representation capabilities. For example, Tran al. [26] recovered missing modalities with cascaded residual autoencoder. Pham al. [20] and Zhao et al. [35] aimed to learn the joint multimodal representation via the cross cross- modality recovery strategy with cycle consistency loss for missing modalities issue. Lian et al. [14] utilized graph neural networks [4] to reconstruct missing modalities to improve MER accuracy.
2.2 Score-based Generative Diffusion Model
The score-based generative model is family of generative models [22-24], which models data distribution through the Stein score function [25, 16]. Given a probability density function p(x), the Stein score function (abbreviated as score) is defined as s(x) Vx ((). As might guess, the score-based generative model aims to parameterize a score network s(x; 0) with learnable parameters 0 and train it to estimate Vx log p(x). Unlike likelihood-based models such normalizing flows [10], score-based models do not have to be normalized and are easier to parameterize.
Perturbing data with a diffusion process. To generate samples with score-based models, we need to consider a diffusion process that corrupts data slowly into random noise. Let {x(t) E IR ) be diffusion process, indexed by the continuous time variable E [0. 77. From the landmark work [24]]
a diffusion process built by a stochastic differential equation (SDE), formally,
(1) where f(., t) denotes the drift coefficient of the SDE, g()) is called the diffusion coefficient, and w represents the standard Brownian motion. Further, the distribution of x(t) is denoted as pr (x(()), such that x0) ~ Po. and xT)) ~ pr- Pa is the data distribution, and pr is the prior distribution.
dx f(x, t t) + g(f)dw,
Generativing samples by reversing the diffusion process. By starting from sample x(T) ~ pr and reversing the diffusion process, can generate a sample from Po- Since the reverse of diffusion process is also a diffusion process [1], running backwards in time and given by the reverse-time SDE:
dx = fxx, t) v(t)²V x p,
px (x)]dt + g(t)dw,
where w is a Brownian motion in the reverse time direction. This reverse-time SDE can be computed once we know the drift and diffusion coefficients of Eq. (1), as well the score for each f [o, 27.
Estimating score via score matching. From the above, it is known that the core of solving reverse- time SDE is obtain the score for each time step The score can be estimated by training time-dependent score network sxx(1), (; on training samples with the score matching [7, 24]:
Lucore E [A (xco B2xx (x)) V Vx(1) Pog (x(() x (00)
(3)
where (0, 7') uniform distribution over [0, 7]. por (x(1) x(0) denotes the transition kernel
from x(0) to x(t), and X(t) denotes a positive weighting function. Finally, we need to know the transition kernel pox (x({) x()) minimize Eq. (3) and optimize s(x(1), 1; The transition kernel is always a Gaussian distribution when the drift coefficient of SDE is affine, where the mean and variance can be obtained with common techniques (see section 5.5 [21]). For example, consider a SDE for score models: dx = /dw, t E [0, 1]. in this case, the corresponding transition kernel is
Po Por (x(1) x() N (5xxxxxx (x)))x(00) 2 2log
( -1))
(4)
When we choose the weighting function X(1) 2 (a2 1), the final objective Lycore
Licore = 1E..=- ((()sl))))))))((((((())))))))))) +
(5)
where z is the random noise sampling from (0,,1)
3

---

3 The Proposed Method
3.1 Prohlem Formulation
Let a tuple (x1, X2, ................................ Xas) denotes M heterogenous modalities of an example, where X is the input of the k-th modality. In the complete case, all modalities are observed and directly fused to facilitate MER. However, in many limited scenarios, there may be some modalities that are unavailable and need to be recovered for better fusion. For simplification, we introduce an indicator a {0, 1 to denote with ag = 0 if the k-th modality is missing, otherwise Qg 1. In the incomplete case, thus, the target is to recover those unobserved modalities Imiss = {kJa = 0} for boosting the downstream tasks. It is worth noting that the missing emotion modalities may not be consistent for each example. To complete the missing modality k E Tmiss: we could sample large amount of data from its latent distribution p(x+), as those advanced generative models do. But the sampling process does not have any semantic guidance, how to generate semantically consistent samples in the sampling process (i.e., avoid semantic ambiguity)? Fortunately, the observed modalities Tobs {klos = 1) contain the semantic information we need, and thus can flavor the sampling process.
Our main idea is to recover the missing emotion modalities Zmiss from their latent distribution spaces on the condition of the available modalities Tohs In other words, the observed emotion modalities Zobs are injected into the recovering process as the semantic conditions to constrain the distribution space of the sampling. Formally, we denote the data distribution of one missing modality as p(xm) (m Imiss) ) and the available ones as p(xz,.), where XI.., is abbreviated for brevity. Hereby, the ultimate goal is to sample missing modal data from the conditional distribution p(Xm| |X,.... We consider a multi-step diffusion model to build this conditional distribution by perturbing Xm. the perturbed conditional transition distribution Pe (Xm ( XT,, (0)) at the 1-th step. Although pe (x))(()(x= (0)) is in general intractable, it can be approximated. We have
Pr (x (t) (T.. (0)) = Pc( (t)xx(())xxn)))) (t), XI.. (0)) (XT.,. (TT. (0)) dx (t)
(6)
= (x(xm(() (t) XL (t)) (XI... (t) XI.... (0)) dx (t)
where px (T, (0)) is tractable because can be derived from the forward diffusion process. the fact that xI.,. (t) is almost the same XI., (0) when t is
In the above approximation, small; t increases, XI... (0) becomes further away from xm in Markov chain so that xm (t) is
almost independent of xI... (0). Motivated by the score-based diffusion model [24], we compute the
score of conditional transition probability in Eq. (6):
V Xm log Pa (Xm XI.. (0)) ~ V xm. log Ep, (XT, (0) (0)) [Pr (xm (t) XI... (t))
(8)
~ V x.. log Pt (xm (t) XT.... (t))
(9)
Xx, log pt [xm (4); XI. (t))),
(10)
where in the second step XI, (t) is a random sample from Pe (XI,, (t) (z) (0)), and note that V xn log Pe (Xm (f) XI,.. (t)) V xm. log Pr (e
(t)) is held because:
Vxa log pr ( (Xm (t); XT. (t))) Vx., log pr (xm XT... (t)) + Vx., log Pr (XI.,. (t)
(11)
V x.. log xm (1) XI... (1)),
(12)
where (t); XI,,, (1) is a vector combination of (t) and XI,,, (t) by well-designed transforma- tion function. This transformation function is called the conditioning mechanism that is described in Sec. 3.4. To infer the gradient, Vx,, log Pe [xm XI,,. (t)]) could be parameterized time-dependent conditioning score network 5m (xm (t), XT.. (t), f; 0m) for modality m. Finally, based on Eq. (5), we derive the conditional score matching objective to be optimized:
cond suooe
[ll (x))m((x((t) XI,.. (t), t; fm)) + + 2
(13)
Given sufficient data and model capacity. the well-trained sm (xmx((t)) (t), XL (t), 12 fm)) can be used to solve the reverse-ting time SDE in Eq. (2) to generate each missing modality Xm. abided by p(xm).
Although the estimated Xm obeys the original distribution, it would deviate from the ground truth when the scatter degree of intra-class samples is large. Thus, we can further refine it with a decoder D, i.e., Xm = Dm (Xm). At the training stage, we minimize the reconstruction error between X,, and Xm. And, we find this refinement step could enhance the performance as shown in Sec. 4.3.

---

3.2 Overview Framework
Shallow Fenture
Missing Modality Diffased Network
Multimodal Fusion Prediction
Longwage
(0
Forward SDE
(T
SIDE
x
Ti
(Langagge Noiie)
(Noise Langwogs)
x
Forward (Viiion Notse)
Reversp-time SDD
x_(0
D.
T.
(Noiee Piston)
Condition: X
dx, (xa(() (30 (4))
Acoustic.....
E
(0)
Forward SDE
(T'
-dx,
sx,x, boo 1:
-D.
To
(Acoustic Notse)
(Noise Acoustis)
Figure 2: The framework of IMDer. Given the input incomplete data (acoustic modality is missed as an example), IMDer encodes shallow feature x; and x, (equal to x, and x, (0) via shallow feature extractor (Sec. 3.3). In missing modality diffused network, we sample x. (T) from prior noise distribution, and then denoise (T') by solving the reverse-time diffusion with score model Sa conditioned on [x]] to generate samples X. (0). Further, X, (0) is fed into the reconstruction module Da to obtain the final recovered modality X, (Sec. 3.4). Finally, X, is combined with x, and X, as complete data for multimodal fusion and prediction to regress the emotional labels. (Sec. 3.5).
The overview framework is illustrated in Fig. 2. It mainly consists of three parts: shallow feature extractor, missing modality diffused network, and multimodal fusion and prediction. Due to the discrepancy of the original dimensional space between modalities, we first align the dimensionality of each modality by extracting multimodal shallow features to facilitate subsequent modality recovery. Then, to alleviate the distribution gap between recovered modalities and original ones, we design missing modality diffused network to model the distribution of each missing modality and sample from the modeled distribution space to generate missing data. At the same time, to reduce the semantic ambiguity of the recovered modalities, the available modalities are embedded as conditions in the recovering process so that we can recover semantically consistent samples from a distribution conditioned on the available modalitics. Finally, to perform the MER, the multimodal fusion and prediction part receives recovered complete multimodal data and uses multimodal transformers to fuse multimodal representation for emotion recognition. The detail is introduced in the next subsections.
3.3 Shallow Feature Extractor
We consider three heterogeneous modalities: language (l), vision (v), and acoustic (a) for the MER task. Since the original dimensional spaces of the three modalities are often inconsistent, they are not suitable for direct use in missing modality recovering. To address this problem, we design a shallow feature extractor that contains three independent encoders &s, {4, v, a}, extract the shallow features of three modalities and map them into the same dimensional space. Each encoder contains a ID convolutional layer. Thus, the subsequent recovery task aims to recover missing modalities from the distribution space of shallow features.
Given input example (aka sequence data), RLXD I and D indicate the sequence length and the feature dimensionality. In the incomplete multimodal case, some modalities are missing either fixedly randomly by guaranteeing least one modality is available in X. For the three modalities mentioned above, total of seven missing combinational cases are included, which are reported in Tab. 1. For convenient statement but without loss of generality, below we recover the missed acoustic modality X as shown in Fig. 2.
extract the shallow features, X' (xx) ). Xx
3.4 Missing Modality Diffuused Network
We consider score network sm to model the distribution of missing modality E Tmiss by perturbing Xm with a common SDE: dx = of o'dw, t E [0, 1], like core-based diffusion [24], the corresponding roverse-time SDE can be derived from Eq. (2) and Eq. (8) as:
dxm 20 sm (Xm, XIass t; 6m dt + o'dw,
(14)
5

---

where XIe, is a condition combining all available modalities. When the score network is well-trained based on Eq. (13), we can generate missing data via Eq. (14). Taking the example that the language modality and vision modality are available, the acoustic modality xa is missing, we build score network 8a conditioned on X3 and x, to recover Xa with the following reverse-time SDE:
dx, - (aa (X) x,), t; Pa)dt +''ww
(15)
where [x] denotes fused feature that is combined of x; and x, by feature concatenation operation and a ID convolution layer with a kernel size of 1. Note that the above feature fusion not required when only one modality is available. To recover the missing modalities using the
above equation, we need to solve the reverse-time SDE with numerical methods. Here, we use
general-purpose Muler-Maruyama numerical method that is based a simple discretization the SDE [24]. When applied to Eq. (15), we can obtain the following iteration rule
xt At) x. + o (xa [x: x. (t),
d'Vato,
(16)
where At is discrete time step size and e (O, I). Given sufficient iteration steps, we can obtain recovered data X,, Subsequently, x, is fed into the reconstruction module to obtain the refined acoustic modality: x Da (X.), where D, denotes the reconstruction module of acoustic modality. For any missing patterns, the set of recovered features can be denoted as Amiss = {&x lk E I thus the reconstruction loss Liee is denoted as:
Σ T
|
X 112.
(17)
Further, optimize the score networks, we can obtain the score matching objective Lucrre under any missing patterns according to Eq. (13):
Lucore
Σ Exxxx N-N.. (itp://nwwooooooll//sssssss///// (x0)) (1,, (4), + 2|[2)-
(18)
By combining free and Luxwre the loss of missing modality diffused network is Lamad = Lree + Laware
Conditioning mechanism. Besides the above, another vital question is how to embed the available modalities (i.e., conditions) into the sample generation. Inspired by cross-modal attention mecha- nism [27], we exploit this mechanism to fuse intermediate representation of xx (1), k: E Imiss and XI... (t) in 8 to achieve condition embedding with flexible manner. In particular, we assume that a certain intermediate representation of (t) and XI... (t) in sy is x (t) and x²_... (t), where x (t) and
x Tab (t) have same tensor size. The cross-modal attention mechanism can be defined as:
QK Vd
Xond (t) softmax
V
(19)
where (1) denotes the condition embedded representation, denotes dimensionality of interme- diate representation. Q = ( K x4__ (1) and V = x1),,(()) (t) where Wo. Wk and Wv are the learnable parameters. Since we consider UNet-style architecture to build the score network, we apply Eq. (19) for condition embedding at each different size of intermediate representation. More details of the score network are elaborated in the supplementary file.
3.5 Multimodal Fusion and Prediction
The recovered data Rmiss = [K E Imiss) and the available data Xats = [xx E Iats), are combined the complete multimodal data for multimodal fusion and prediction of emotions. We employ multimodal transformers Th [27] to fuse complete multimodal data &miss U Xobs and the fused feature is used to predict emotions by fully connected layers. We integrate the above losses to reach the full optimization objective:
Luaa = Cousk + 3.Cmmdd
(20)
where Lusk is the task loss defined as mean absolute error, 33 controls the importance of different loss functions. The entire optimization implemented in an end-to-ent manner, and the concrete training about the configuration of incomplete modalities could be found in the experiment part.

---

Experiments
4.1 Datasets and Implementation Details
Datasets. We consider two standard MER datasets to conduct experiments, including CMU-MOSI [32] and CMU-MOSEI [33]. CMU-MOSI consists of 2199 monologue video clips. Where 1284, 229, and 686 samples are used as training. validation, and testing set. CMU-MOSEI contains 22856 samples of movie review video clips. Where 16326 samples are used for training, the remaining 1871 and 4659 samples are used for validation and testing. On the two datasets, extract the language features via pre-trained BERT model [9] and obtain a 768-dimensional hidden state as the word embedding. For vision modality, each video frame encoded via Facet [8] to represent the presence of the total 35 facial action units. The acoustic modality was processed by COVAREP [5] to obtain the 74-dimensional features.
Evaluation metrics. CMU-MOSI and CMU-MOSEI are collections of videos from YouTube of people providing opinions, that have different human annotations, including human emotion score and emotion categories (only for CMU-MOSEI). Our work focuses on human emotion score. Each sample in two datasets was labeled with human emotion score that ranges from -3 to 3, including highly negative, negative, weakly negative, neutral, weakly positive, positive, and highly positive. To make comprehensive comparison, we evaluate the performance using the following metrics: 7-class accuracy (ACC,), binary accuracy (ACC2), and Fl score.
Implementation details. Following [20,35 35, 14], we investigate the performance of different methods under two commonly- lused protocols: (1) fixed missing protocol. (2) random missing protocol. For the fixed missing protocol, we consistently discard one modality (i.e., {1, v), {1, a), {v, a}) or two modalities (i.e., {1). {v). (a)). For the random missing protocol, the missing patterns are randomized for each sample (i.e., one or two modalities may be missing for each sample). Here, we use the missing >> m;
rate (MR) to measure the overall missingness of the dataset. The MR is defined as MR = 1-2
NxM
where m, denotes the number of available modalities for 1th sample, N denotes the total number of samples, and M indicates the number of modalities. We also ensure that at least one modality is available for each sample, so my 2 1 and MR < M-1, For three modalities,
choose the MR from [0.0, 0.1, 0.7]. where 0.7 is an approximation of M-1 with the same meaning. We keep the same MR during training, validation, and testing phases, consistent with previous work [14]. The optimal setting for 33 is set to 0. via the performance on the validation set. run each experiment five times and report the average values on the testing set.
Table 1: MER accuracy comparison under fixed missing protocol. The values reported in each cell denote ACCs/FI/ACC, Bold is the best.
Datasets Available
DCCA
DCCAE 29)
MCTN 20
MMIN [5]
GCNe [14]
IMDer (Ours)
73.673387303
76.47
79.1779.
83.8783.8
4/6
63.7783.6728
$4.8734.77 44.8
47.7/41.5/166
52.6/511/17.
55.0/54.4
16.
57.0
61.3/ 60.8 22.2
[0]
50.5/46.1/16.
56.1/54.5/16
16.5
55.3/
/6.1/54.5// 16.6
/2.0/62.2/ 22.0
[1,0]
74.9/550/30.3
76.7/76.8/
30.0
81.1/81.//2.
83.8
0..8/83.//4...
84.3/84.2/4.44
85.5 /
MOSI
[La)]
74.7/74.8/2.7
77.0/77.9/ 30.2
8L0/8L0/43.
84.0
84.0
84.5/
/5.4/85.3// 85.3/
[F.a]
54.0/ 17.4
16.8 57.5
68.4/58.5
62.0/61.//72
63.6 /3,6//3.4/// 23,8
(1,0,0)
75.3/75.44
77.3/77.4/31..
814/815/43.4
84.6/8444/444
85.2/85.1
85.7 05.77885.6455
Averuge
63.9/61.9/0000 25.2//.9//2000061//688///22 66.1/488/44
69.2/69.9/3.3
/2.7/714/316
/3.1/72.8/32.
75.5
78.3778.7746.7
8267828750
82.3782
83.0783.751.22
14..7844.55
(v)
61.9/55.7/4.3
/ 40.1
62.6/571//4.6
/9.3//6.0//
40.7
61.9/61.//11.
63.9//6..// 1.3.6/440 42.6
(a)
62.0/ 50.2 41.1
31.4/53.8/-4
/2.7
58.9
28.9//9.5/44 40.4
60.2 /4..
63.8 60.6 41.7
MOSEI
(1,0)
80.3 79.7
K1./8332/550
50.4
84.3/84.4/51.
/5.0/85.0//
(L.a)
79.2//9.22/667
80.0 80.0 47.4
83.5/8333/0077
83.7/
84.3/84.4/51.
85,1 15.11/8.1/// /5,1
[1.4]
(5.4/56.9/40 41.5
62.7 59.2/44
63.7/62.7/421
63.5/61.//413
/4.1/
63.5 42.8 64.9
80.7 47.7
81.2
48.2
84.2/84.2/51.2
84.2/44.//2.4
85.2/85.1/513
85.1
/
Averuge
72.3 68.8 44,5
72.4 70.2 /0.2
44.6
72.5
73.5/73.5147
14.7/73.7//7.
76.0 75.3/ 48.5
4.2 Comparison with the state-of-the-arts
We compare IMDer with the current state-of-the-art incomplete multimodal emotion recognition methods based on modality recovery mechanism, including MCTN [20], MMIN [35], GCNet [14]. For comprehensive comparison, consider two deep learning- baaning-based non-recovery methods with
7

---

canonical correlation maximization DCCA [2] and DCCAE [29] in incomplete multimodal learning conduct comparisons. Below, report the quantitative and qualitative experimental results.
Table 2: MER accuracy comparison under random missing protocol. The values reported in each cell
denote ACC//FL/ACC, MR: Missing Rate. Bold the best.
Datasets
MR
DCCAT2]
DCCAE 29
MCTN 220
MMIN [35]
GCNet [14]
MMDer (Ours)
15.37 30.5
71.37774331
84.678444755
35278517444 352
85.7 75.678557755
0.1
72.1/722// 28.0 74.5/74.7/28
78.4/78.5/ /9.8/999
84,9/ 84,8 44.8
69.3/69.1/288
718/7119/266
75.6/75.7/885
79.0/79.138.9
79.4/79.5/40..
33.5/83./4 44.3
65.4/65.2/25.7
67.0/66.7/255
71.3/71.2355
76.1/76.2/369
77.2/77.2/38.
81.2/ 31.00/2222
MOSI
0.4
62.8/62.//4.
63.6/62.8/42
68.0/67.//29
71.7/716/349
74.3/74.443
78.6
78.5 39.7
60.9/59.9/266 62.0/61.3/230
65.4/64.//122
67.2/66.5/3222
70.0/
76.2 75,9 37.9
0.6
59.6/
63.8/225
24.9/6..//9.11
67.7/66.
74.7 / 74.0 35,8
0.7
37.4/56.00/ 20.4
58.1//7.4/2066
51.2/
628/610/284
65.77
71.9
Averuge
64.2/
24.8 66.7/66.3/
70.1 34.8
73.5/73.1
75.2
79.6/79.31
0.0
80.7 / 47.
812/8112/48
85.27
85.1 85.1 53,4
77.4/77.3/ 46.2 78.47783/440
8L8/8L///
819/8113 82.3/82.1/51 5 84.8 84,6 53.1
0.2
73,8 74.00/4. 45.1
75.5/75.4/46.3
78.07 48.6
79.8 /7.8
/
/
82,4/ 52.0 82.7
71.1/71.2/366
72.3/72.2/456
76.
76.9
77.2/75.5/41 78.1
77.5
76.88 49.2
/ 80.7 51.3
MOSEI
69.4/6994/43)
70.3/7000/44.0
74.3/
75.2/7226/47.5
76.0/74.9/
48.0
79.3
78.1 50.0
0.5
6 42.5
73.6/72.6/
73.9 /46.7
74.9/
46.7 79.0
77.4 49.2
66.2/ / 42.4
67.6/63.2/423
73.2/7L1/433
73.2/70.3
/
74.1/
45.1
78.0
75.5 48.5
65.6/61.0/42 42.1
66.6/62.6
442.5 72.7
43.6
73.1
/4.8 7
44.5
77.3
74.6 47.6
Averuge 71.5 / 70.3 44.1
77.0/ 46.9 77.3/55./448
77.9/ 48.3
80,9 79.8 / 50.6
Quantitative results. Tab. and Tab. 2 list the quantitative results of fixed missing protocol and random missing protocol on two datasets, respectively. We have the following observations:
1. IMDer achieves the best results on the two MER datasets under both fixed missing protocol and random missing protocol, demonstrating superiority of IMDer. Compared with the non-recovery- based methods [2, 29]. IMDer obtains consistent MER performance gains. This be explained that IMDer can explicitly recover the missing modalities which can provide extra complementary information for MER. Compared with the recovery-basee methods [20, 35, 14], the consistent improvements of IMDer indicate the superiority of maintaining distribution consistency between the recovered and the original modalities.
Avail:; Miss: V.A
AvailyV
Miss:: L.A
Avail:A Miss: L.
MCTN
MMIN
GCNet
Ours (8x)
Ours (x₂)
Ground Truth
Recovered
Figure 3: Visualization of recovered modalities. Avail. means available. The distribution of modality recovered by our method is much closer to the ground truth than the compared methods.
From the experimental results under fixed missing protocol in Tab. 1, we can observe that when language modality is missing, our proposed IMDer obtains more obvious enhancements. This is reasonable because language modality is crucial for robust MER, i.e., is highly abstract and semantic and contains more discriminative information [20]. Thus, it is vital to complete the missing language modality for robust MER. Compared with other recovery-bases methods, our proposed IMDer explicitly models the distribution space of the language modality and recovers the distributionally and

---

semantically consistent language features to facilitate robust MER. The comprehensive comparison in Tab. 1 verify the effectiveness of our method.
3. Compared with other MER methods, IMDer suffers from less performance degradation as the modality missing rate increases. This can be verified in Tab. (random missing protocol). On CMU- MOSI, when the missing rate increases from 0.0 to 0.7, the ACC₂ of the compared recovery-based methods declines 19.5% 21.8% while our IMDer merely declines 13.8%. For the CMU-MOSEI, as the missing rate increases from 0.0 to 0.7, ACC₂ of other recovery-based methods show obvious decline: 11.2% 12%. As comparison, our proposed IMDer shows 7.8% accuracy degradation. These comparisons demonstrate IMDer's generalizability under random missing protocol.
Qualitative results. Fig. 3 visualizes the distribution of recovered data and ground truth for different recovery-based methods under the fixed missing protocol. We randomly select 500 samples in the testing set from the CMU-MOSHI dataset. The features of the selected samples projected into 2D space by t-SNE [28]. From these results, we can observe that the distribution between the original and recovered modalities estimated by IMDer is much closer than other methods, indicating IMDer effectively mitigates the distribution gap between recovered and the original data.
Table Ablation study the key components in IMDer under the average random missing protocol.
Datasets
8k
D
ACC₂
ACC7
79.6
79.3
40.5
CMU-MOSI
X
77.8
77.7
38.4
75.8
75.6
36.8
X
74,5
74.4
36.0
80.9
79.8
50.6
79.8
78.5
49.4
CMU-MOSEI
78.0
76.9
48.8
76.5
76.0
48.5
Unconditional Recovering
Conditional
Recovering
Highty Negative
A Negative
Weakly Negathve
Neutral
Wenkdr Posiivv
O Positive
? Highhy Positive
Ground Truth
Recovered
Figure 4: Visualization of the unconditional and conditional modality recovering. Conditional recovering (bottom row) shows the promising emotion category separability, indicating that using the vanilla available modalities as conditions can reduce the semantic ambiguity.
4.3 Ablation study
Quantitative analysis. We evaluate the effects the key components IMDer, including score model and reconstruction module Dr. The results are illustrated in Tab. 3, we conclude the conclusions as: 1) Recovering missing modalities by reconstruction module Dj or score model 8), effective. It is because explicitly recovering the modality provide meaningful complementary

---

information. Further, we find that the score model- based modality recovery can provide more performance improvements. This because the score model can recover data from the distribution of each missing modality, and the distribution between recovered data and original data is more consistent. 2) Combing 85: with Dr brings further benefits, which proves the refinement mechanism provided by Dj and the diffusion-based score model 85 can benefit each other mutually.
Visualization of unconditional/conittonna recovering. Fig. visualizes the recovered and the original modality under the unconditional and conditional settings. Specifically, we randomly select 70 samples (ten samples for each class) in the testing set of the CMU-MOSEI dataset. The features of the selected samples are projected into a 2D space by t-SNE [28]. We can observe that unconditional recovering suffer from semantic ambiguity due to the lack of semantic guidance. In contrast, the data recovered by conditional recovering has been clustered and can be distinguished across seven different emotion classes, indicating that conditional recovering can mitigate semantic ambiguity between the recovered and the original modalities.
5 Conclusion and Discussion
In this paper, we try to challenge MER under incomplete multimodalities by proposing an Incomplete Multimodality-Diffuss
emotion recognition (IMDer) method. To recover the missing modalities, IMDer exploits the score-based diffusion model that maps the input random noise into the distribution space of missing modalities and recovers missing data abided by their original distributions. To reduce semantic ambiguity between the missing and the recovered modalities, the available modalities used conditions to guide the diffusion- recovering process. Quantitative and qualitative experiments consistently demonstrate the effectiveness of IMDer.
Limitations. This work also has some limitations. The first limitation is that solving the reverse- time SDE incurs high computational overhead and may not be suitable for scenarios requiring real-time performance, and we will try mitigate this issue in future work, such as replacing the Euler-Maruyama solvers with ordinary differential equation solvers. In addition, our work may not work to recognize emotion in other scenarios such as human-robot interaction, because emotion signal might change depending on the context, culture, etc.
Ethical implications discussion. Recognizing human emotion from multimodalities helps build better human numanecomputer interactions. Albeit the obvious benefits, this technology also introduces some potential negative societal impact. For example, multimodal data contains a large number of faces, facial data will bring potential social privacy and security issues.
Acknowledgement
This work was supported by the National Natural Science Foundation of China (Grant Nos. 62072244, 62102180), the fundamental research funds for the central universities (Grant No. 3091901 1232), the Natural Science Foundation of Shandong Province (Grant Nos. ZR2020LZHO0S ZR2022LZH003), the Natural Science Foundation of Jiangsu Province (Grant No. BK20210329), Shuangchuang Program of Jiangsu Province(Grant No. JSSCBS20210200), and in part by State Key Laboratory of High-end Server & Storage Technology.
References
[1] Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications,
12(3):313-326, 1982.
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In
International conference on machine learning, pages 1247-1255. PMLR, 2013. 3, 7,8
Qiong Cai, Hao Wang. Zhenmin Li, and Xiao Liu. survey on multimodal data-driven smart healthcare
systems: approaches and applications. IEEE Access, 7: 7:133583-133599 2019.
[4] Zhen Cui, Ling Zhou, Chaoqun Wang. Chunyan Xu. and Jian Yang. Visual micro-pattern propagation.
IEEE Transactions on Puttern Analysis and Machine Intelligence, 45(1):1267-1188, 2022.
[5] Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. Covarep a collabora- tive voice analysis repository for speech technologies. In 2014 ieee international conference on acoustics, speech and signal processing (icassp), pages 960-964. IEEE, 2014.
[6] Harold Hotelling. Relations between two sets of variates. Breakthroughs in statistics: methodology and distribution, pages 162- 190, 1992.
10

---

[7] Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching.
Journal of Machine Learning Research, 6(4), 2005. 3
[8]
iMotions. Facial expression analysis. https: /tpp:///mmon //tmotions......mm cow/ wtt.ccoo///puuue http://duuttt/moooooss-ssooo 1ab/mmuullammmemeee
fea-facial- expreasion- analysis/ 2017.
[9] Jacob Devlin Ming-We Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, In Proceedings of naacL-HLT, pages 4171-4186, 2019. 7
[10] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions, Advances in neural information p processing systems, 31, 2018.
[11] Elsa A Kirchner, Stephen Fairelough, and Frank Kirchner. Embedded multimodal interfaces in robotics: applications. future trends, and societal implications. In The Handbook of Mulimmoda-Multiiensor Language Processing. Software, Commerciatization, and Emerging Directions- Volume 3, pages
Interfaces:
523-576, ACM, 2019. 1
[12] Michelle A Lee. Yuke Zhu, Krishnan Srinivasan, Parth Shah, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bobg. Making sense of vision and touch: Self-supervised learning of multimodal representations
contact-rich tasks. 2019 International Conference on Robotics and Automation
(ICRA), pages -954-0 IEEE, 2019.
[13] Yong Li, Yuanzhi Wang, and Zhen Cui. Decoupled multimodal distilling for emotion recognition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. I [14] Zheng Lian, Lan Chen, Licai Sun, Bin Liu, and Jianhua Tao. Genet: Graph completion network for incomplete multimodal learning in conversation, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. 2, 3, 7,8
[15] Tao Liang, Guosheng Lin, Lei Feng, Yan Zhang, and Fengmao Lv. Attention is not enough: Mitigating the distribution discrepancy in asynchronous multimodal sequence fusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8148-8156, 2021.
[16] Qiang Liu, Jason Lee, and Michael Jordan. A kernelized stein discrepancy for goodness-of-it tests. In International conference on machine learning. pages 276-284. PMIR, 2016. 3
[17] Fengmao Lv. Xiang Chen. Yanyong Huang, Lixin Duan, and Guosheng Lin. Progressive modality reinforcement for human multimodal emotion recognition from unaligned multimodal sequences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2554-2562, 2021.1 1
[18] Ghulam Muhammad, Fatima Alshehri, Fakhri Karray, Abdulmotaleb El Saddik, Mansour Alsulaiman, and Tiago H Falk. A comprehensive survey on multimodal medical signals fusion for smart healthcare systems.
Information Fusion. 76:355-375 2021.
[19] Srinivas Parthasarathy and Shiva Sundaram. Training strategies handle missing modalities for audio- Companion Publication of the 2020 International Conference on Multimodal Interaction, pages 400-404, 2020.
visual expression recognition.
[20] Hai Pham, Paul Pu Liang, Thomas Manzini, Louis- Moreney, and Barabáss Póczos. Found in translation: Learning robust joint representations by cyclic translations between modalities. In Proceedings
of the AAAI Conference Artificial Intelligence, volume 33, pages 6892 6899. 2019. 2, 3, 7,8
[21] Simo Särkkä and Arno Solin. Applied stochastic differential equations, volume 10. Cambridge University
Press, 2019. 3
[22] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution.
Advances in neural information processing systems, 32, 2019. 3
[23] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances
in nearal information processing systems, 13:12448-12448. 2020.
[24] Yang Song. Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-hased generative modeling through stochastic differential equations. In International Conference on
Learning Representations, 2021. 2, 3, 4, 5, 6
[25] Charles Stein. A bound for the error in the normal approximation to the distribution of a sum of dependent random variables. In Proc. Sixth Berkeley Symp. Math. Stot. Prob., pages 583-602, 1972. 3
[26] Luan Tran, Xiaoming Liu, Jiayu Zhou, and Rong Jin. Missing modalities imputation via cascaded residual autoencoder. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1405-1414, 2017.
[27] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philipen
Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference, Association for Computational Linguistics. Meeting, volume 2019, page 6558. NIH Public
Access, 2019. 1, 6
[28] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning
research, 9(11), 1), 2008, 9, 10
[29] Weiran Wang, Raman Arora, Karen I Livescu, and Jeff Bilmes. On deep multi-view representation learning.
In International conference on machine learning, pages 1083-1092. PMLR, 2015. 3, 7,8 8
[30] Yuanzhi Wang. Zhen Cui. and Yong Li. Distribtion-coosiisent modal recovering for incomplete multimodal learning. In Proceedings of the IEEE/CVF Intermational Conference on Computer Vision (ICCV), pages
October 2023.
[31] Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Tensor fusion network for multimodal sentiment analysis. In Empirical Methods in Natural Language Processing.
EMNLP, 2017. 1
11

---

[32]
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems, 31-6):82-88. 2016. 7
[33] AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis PhillipPi Morency, Multimodal language analysis the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 2236-2246, 2018. 7
[34] Changqing Zhang, Yajie Cui, Zongbo Han, Joey Tianyi Zhou, Huazhu Fu, and Qinghua Hu. Deep partial multi-view learning. JEEE transactions on pattern analysis and machine intelligence, 44(5):2402-2415,
2020. 3
[35] Jinmiing Zhao, Ruichen Li, and Qin Jin. Missing modality imagination network for emotion recognition with uncertain missing modalities. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing
(Volame 1: Long Papers), pages 2608 2618, 2021. 2, 3. 7, 8
[36] Sicheng Zhao, Guoli Jia, Jufeng Yang. Guiguang Ding, Kurt. Keutzer. Emotion recognition from multiple modalities: Fundamentals and methodologies. IEEE Signal Processing Magazine, 38(6):59-73,
2021.
12


Reviews:
[{"review_id": "ZhKIl0FXbt", "reviewer": "Reviewer_tA7f", "summary": "This paper proposed a diffusion-based modality recovery mechanism to generate the missing modalities and then fuse the available modalities and the generated missing modalities for emotion prediction. The experiment results on MOSI and MOSEI shows that the proposed method can achieve state-of-the-art performance. However, there are some errors when introducing the related works, where the MCTN and MMIN aim to learn the multimodal joint representation through the cross-modality prediction strategy for missing modalities problems, not use the predicted modality for emotion prediction.", "questions": "Directly predicting the missing modalities signal or the sequence features of the missing modalities is very difficult based on the downstream dataset which only has thousands of samples, so the question is how to solve it and how is the prediction performance of missing modalities (for example, MSE/RMSE Loss).", "limitations": "As Weaknesses.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper proposed an efficient missing modality prediction method for missing modalities problems and reach the state-of-the-art performance.", "weaknesses": "1. There are some conceptual errors in this paper, shown in Summary Block.\n2. The MOSI and MOSEI dataset are very similar, the author should explore more different datasets to verify the effectiveness of the proposed methods. \n3. In Figure 4, the meaning of the different columns is confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Thanks for recognizing our innovation and promising results. Below, we respond to each Question (Q) with an Answer (A).\n\n**Q1: There are some conceptual errors in this paper, shown in Summary Block.**\n\nA1: For your question about some statement in the related work, we carefully check this part, it might exist some unclear statement and should not be conceptual errors. According to your suggestion,  we clarify the descriptions w.r.t. MTCN and MMIN: the MCTN and MMIN aim to learn the joint multimodal representation via the cross-modality recovery strategy with cycle consistency loss for missing modalities issue.\n\n**Q2: The MOSI and MOSEI dataset are very similar, the author should explore more different datasets to verify the effectiveness of the proposed methods.**\n\nA2:  Thanks for your suggestion. MOSI and MOSEI datasets are wildly used for multimodal emotion recognition. MOSI focuses on movie reviews and MOSEI contains more than  250 topics. For your concern, we conduct extra experiments on another Chinese MER dataset CH-SIMS [a]. The experimental results are shown in the following table. Obviously, our proposed IMDer consistently achieves better results than the previous best methods: MMIN and GCNet under random missing protocol (MR means missing rate).\n|MR|MMIN|GCNet|Ours|\n|:----|:----|:----|:----|\n| |ACC$_2$/F1/ACC$_5$|ACC$_2$/F1/ACC$_5$|ACC$_2$/F1/ACC$_5$|\n|0.0|74.9/75.0/45.3|**76.7**/**76.8**/50.1|76.3/76.4/**50.7**|\n|0.1|73.4/73.2/44.7|75.2/75.3/48.0|**75.4/75.5/50.1**|\n|0.2|71.9/71.5/44.1|73.7/73.6/45.1|**74.7/74.5/47.3**|\n|0.3|70.5/70.0/40.3|72.3/72.5/43.2|**74.2/73.6/45.6**|\n|0.4|69.6/68.9/39.0|71.1/71.2/41.0|**73.7/73.3/44.6**|\n|0.5|67.1/67.1/37.0|69.7/69.4/40.5|**72.7/72.3/43.3**|\n|0.6|66.7/65.6/37.2|68.9/68.9/39.8|**71.3/69.8/42.4**|\n|0.7|64.3/63.8/35.1|67.8/67.8/35.9|**69.8/69.6/42.2**|\n|Average|69.8/69.4/40.3|71.9/71.9/43.0|**73.5/73.1/45.8**|\n\n[a]  \"Ch-sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality.\" ACL. 2020.\n\n**Q3: In Figure 4, the meaning of the different columns is confusing.**\n\nA3:  In Figure 4, each column illustrates a visualization comparison between unconditional (top row) and conditional (bottom row) recovery mechanism when a certain modality is missing. The missing modality for the first/second/third column is language/vision/acoustic modality, respectively. From Figure 4, we can conclude that conditional modality recovery mechanism shows the promising emotion category separability, indicating that using the available modalities as conditions can effectively reduce the semantic ambiguity. We will re-clarify the explanation in the revision.\n\n**Q4: Directly predicting the missing modalities signal or the sequence features of the missing modalities is very difficult based on the downstream dataset which only has thousands of samples, so the question is how to solve it and how is the prediction performance of missing modalities (for example, MSE/RMSE Loss).**\n\nA4: To reduce the difficulty and ambiguity of modality recovery, we conduct modality reconstruction in the low-dimensional feature distribution space. In addition, we pay more attention to the semantic consistency while recovering the modality, rather than just minimizing the MSE.  As to measure the prediction performance of missing modalities, in addition to the objective loss (i.e., score matching objective and MSE Loss), we have adopted 1) the accuracy of the downstream MER (Please see Table 1 and 2); 2) distribution visualization results for the recovery modality (Please see Figure 3 and 4).\n\n"}, {"review_id": "oYokQTc6iw", "reviewer": "Reviewer_4zZs", "summary": "The paper presented an Incomplete Multimodality-Diffused emotion recognition (IMDer) method to mitigate the challenge of MER under incomplete multimodalities. To recover the missing modalities, IMDer exploited the score-based diffusion model that mapped the input Gaussian noise into the desired distribution space of the missing modalities and recovered missing data abided by their original distributions.", "questions": "1.  Sec. 3.1 contains too much theory analysis. It is suggested to simplify or remove part of them to shorten the section.\n2. In Figure 2, there is no connection between D_t and D_v. That may take confusion in understanding. Moreover, from D_a to hat(X)_a, there is no definition, which also leads to confusion.\n3. In section 4.1, the Dataset part includes too much information. It is suggested to separate them into multiple parts which contain independent information, e.g. Datasets, Evaluation metrics….\n4. In Table 2, there is MR: Missing Rate, but the missing rate involves which modality? \n5. Some errors,\n1) Line 185, with an common SDE…\n", "limitations": "See weakness.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed approach IMDer is applied to recover missing modalities.\n2. The algorithm explanation contains a clear and detailed theory analysis.\n3. Sufficient experiments and results. \n", "weaknesses": "1. Motivation is not clear for the proposed solution.\n2. Major contributions are not well summarized.\n3. Sec. 3.1 is too long, consider separating theory analysis into other sections.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Thanks for recognizing our innovation and promising results. Below, we respond to each Question (Q) with an Answer (A).\n\n**Q1: Motivation is not clear for the proposed solution.**\n\nA1: The performance of MER in controlled scenarios has achieved satisfactory results in recent years. However, in real-world scenarios, not all modalities are consistently available, e.g., language data may be missing due to speech recognition errors; video data may be inaccessible due to privacy  and security concern. Ultimately, these incomplete multimodal data significantly hinder the performance of MER. Our main motivation is to preserve the performance of MER under missing modality cases as much as possible by recovering distributional and semantically consistent modality.\n\n**Q2: Major contributions are not well summarized.**\n\nA2: Our major contribution is to recover distribution-consistent missing modalities by the score-based diffusion models. In particular, to reduce the semantic ambiguity between the missing and the corresponding recovered modalities, we use the vanilla available modalities as the semantic conditions to guide and refine the recovering process. We will re-clarify this in the revision.\n\n**Q3: Sec. 3.1 contains too much theory analysis. It is suggested to simplify or remove part of them to shorten the section.**\n\nA3: Thanks for the suggestion. We will reorganize the theory analysis part by shortening this part in the revision.\n\n**Q4: In Figure 2, there is no connection between D_t and D_v. That may take confusion in understanding. Moreover, from D_a to hat(X)_a, there is no definition, which also leads to confusion.**\n\nA4: In Figure 2, we take the example that the acoustic modality is missing, thus there should not be connection between the two available modalities D_l (not “D_t”, typo by the reviewer) and D_v. In the caption of Figure 2, we had defined D_a to hat(X)_a: “Further, $\\overset{\\sim}{\\text{x}}\\_{a}(0)$ is fed into the reconstruction module $\\mathcal{D}\\_{a}$ to obtain the final recovered modality $\\hat{\\text{x}}\\_{a}$”. For your concern, we will add the definition from D_a to hat(X)_a in Figure 2 for clarity.\n\n**Q5: In section 4.1, the Dataset part includes too much information. It is suggested to separate them into multiple parts which contain independent information, e.g. Datasets, Evaluation metrics….**\n\nA5:  Thanks for your suggestion. In the next revision, we will separate Dataset part into Datasets and Evaluation metrics parts for clarity.\n\n**Q6: In Table 2, there is MR: Missing Rate, but the missing rate involves which modality?**\n\nA6: MR measures the overall missingness ratio of the dataset, where the missing patterns are randomized for each sample, please see the description in lines 243-251.\n\n**Q7: Some errors, Line 185, with an common SDE…**\n\nA7: Thanks for your suggestion. We will polish the full paper and correct typos and grammar errors, including the article “a” in “with an common”."}, {"review_id": "cVvuy4Nsp1", "reviewer": "Reviewer_bs2t", "summary": "Paper proposes IMDer for multimodal emotion recognition under missing modalities. Language, vision, and audio modalities are evaluated using the CMU-MOSI and CMU-MOSEI datasets. Quantitative and qualitative results are shown including comparisons to state of the art, visualizations, and ablation study.", "questions": "For the video modality, what exactly are the 35 AUs?", "limitations": "Limitations are adequately addressed.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "A nice approach to MER under missing modalities is proposed. The proposed approach is well motivated and detailed in paper. Contributions and extensions to state of the art are well defined in paper.\n\nEvaluation is well conducted on two public datasets, showing encouraging results compared to state of the art. Fixed missing and random missing protocols are well done. Ablation study shows utility of both score model and reconstruction module. Qualitative results (Figure 3) look good showing closer distribution to ground truth compared to other approaches. ", "weaknesses": "It is not clear how AUs are used as the video modality. OpenFace does not detect 35 different AUs. Is the AU feature vector AU occurrence and intensity? If so, does the occurrence add anything of value to it? An intensity of 0 would indicate the AU is not active. \n\nMinor: There are typos that need fixed (e.g., \"languate\" on line 270).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Thanks for recognizing our innovation and promising results. Below, we respond to each Question (Q) with an Answer (A).\n\n**Q1: It is not clear how AUs are used as the video modality. OpenFace does not detect 35 different AUs. Is the AU feature vector AU occurrence and intensity? If so, does the occurrence add anything of value to it? An intensity of 0 would indicate the AU is not active.**\n\nA1:  We apologize for the misunderstanding caused by the literature citation error that comes from the previous literature [a][b]. The literature [a][b] had been also quoted in our manuscript. For the question, we make a careful retrieval about the source, and further make the clarification as follows:\n\nThe Facet is from iMotions [c] which is a commercial software toolkit not from OpenFace. Concretely, the vision modality we used is referred to the prior work [d] and is extracted by the CMU Multi-camp team (https://github.com/CMU-MultiComp-Lab/CMU-MultimodalSDK). For a video frame, the Facet is used to indicate 35 AUs that record the facial muscle movement for representing per-frame basic and advanced emotions (please see page 12 of [d]). For example,  for one of the face frames of a video clip, we extract the predicted values of 35 AUs that are the float value and not projected by activation function (i.e., the AUs value is not 0 or 1) as the feature vector of the face. If the video clip has 50 frames, then the tensor size of this vision modality is $50\\times 35$.\n\n[a] \"Decoupled Multimodal Distilling for Emotion Recognition.\" CVPR, 2023.\n\n[b] \"Attention is not enough: Mitigating the distribution discrepancy in asynchronous multimodal sequence fusion.\" ICCV, 2021.\n\n[c] iMotions 2017. Facial expression analysis. [https://imotions.com/products/imotions-lab/modules/fea-facial-expression-analysis/](https://imotions.com/products/imotions-lab/modules/fea-facial-expression-analysis/)\n\n[d] \"Multimodal transformer for unaligned multimodal language sequences.\" ACL, 2019.\n\n**Q2: Minor: There are typos that need fixed (e.g., \"languate\" on line 270).**\n\nA2: We will fix some typos in the revision according to your valuable suggestion.\n\n**Q3: For the video modality, what exactly are the 35 AUs?** \n\nA3: We use Facet from iMotions [c] to extract 35 facial action units (AUs) that record the facial muscle movement for representing basic and advanced emotions. The detailed definition of facial AUs may be obtained by decoding the Facet software. Here we follow the prior work [d] and the study (https://github.com/CMU-MultiComp-Lab/CMU-MultimodalSDK) of the CMU Multi-camp team. Given one of the face frames of a video clip, we extract the predicted values of 35 AUs that are the float value and not projected by activation function (i.e., the AUs value is not 0 or 1) as the feature vector of the face (more details are provided in A1)."}, {"review_id": "VZkcbOMi2H", "reviewer": "Reviewer_BscJ", "summary": "This paper proposes a diffusion model-based generation method to solve the problem of missing modes in multimodal emotion recognition (MER), and in the model, the available modalities are utilized as prior conditions to guide and refine the recovering process to reduce the semantic ambiguity between the missing and the corresponding recovered modalities. Besides, quantitative experimental results verify that IMDer obtains state-of-the-art MER accuracy under various missing modality patterns.\n", "questions": "Besides the above weakness, I also have the following questions:\nIs it reasonable and valid for solving reverse-time SDE with some approximate scores? Does it limit the effect of the model?", "limitations": "No.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "# Originality  \nThe authors use a score-based diffusion model to address the modal missingness in MER while using a cross-attention mechanism to guide and refine the recovery process by using other available modalities as a priori conditions to ensure that the recovered modalities are consistent in distribution and semantics. \n\n# Quality\nIMDer performs relatively well on each dataset, while the framework is neat and clear, making it a relatively effective and inspiring method.\n\n# Clarity \nThe general idea and framework of this paper are relatively clearly described, but some of the simplifications in the formulas are not clearly described.\n\n# Significance\nAlthough the IMDer proposed in the article reaches SOTA, it is more significant to the community for two main reasons: \n1. IMDer can not only be used to solve the modal deficiency problem but also be inspiring for other fields such as multimodal transformation and fusion.\n2. It provides a mathematical derivation of the diffusion model for generating missing modalities.\nOverall, the contribution of IMDer is significant.\n", "weaknesses": "Although this paper has achieved relatively good results, it cannot deal with the problem of intra-modal missing, e.g. some parts of the audio is missing. In addition, the ablation experiments are not sufficiently done, and other modules such as the semantic disambiguation module are not subject to ablation experiments. There is insufficient explanation of whether the simplification of the derivation of the formula is reasonable.\n\nThe experiments in the paper are insufficient, especially the ablation experiments, and the model comparison should be compared with the underlying methods cited in the Multimodal Fusion and Prediction module to demonstrate the effectiveness of the diffusion module. \n\nThe effectiveness of the semantic disambiguation part of diffusion is not evaluated, and the improvement of the recognition performance of Incomplete Multimodality cannot be assessed. It is not clear whether the improvement comes from the semantic information provided by other modalities during diffusion or rather from the extraction of better intra-modal information features.\n\nSome closely related surveys and reviews on MER are missing, such as \"Multimodal machine learning: A survey and taxonomy\", and \"Emotion Recognition From Multiple Modalities: Fundamentals and methodologies\".\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Thanks for recognizing our innovation and promising results. Below, we respond to each Question (Q) with an Answer (A).\n\n**Q1: Although this paper has achieved relatively good results, it cannot deal with the problem of intra-modal missing, e.g. some parts of the audio is missing.**\n\nA1: The problem of intra-modal missing is a sequence or feature recovery task, and the objective is different from modal missing that we focus on. We are also interested in the problem of intra-modal missing and consider combining it with the modal missing problem by multi-task learning.\n\n**Q2: In addition, the ablation experiments are not sufficiently done, and  other modules such as the semantic disambiguation module are not subject to ablation experiments.**\n\nA2:  The ablation studies on possible cases have actually been provided in Section 4.3 in the manuscript and the supplementary material, including the semantic disambiguation. Concretely, we show the visualization results of the ablation experiments for semantic disambiguation (i.e., the conditional recovering) in Figure 4, and report the quantitative results of the ablation experiments for semantic disambiguation in Section 4 in the supplementary materials.\n\n**Q3: There is insufficient explanation of whether the simplification of the derivation of the formula is reasonable.**\n\nA3: There are two simplifications in our Problem Formulation. The first (in Eq.7) is \n$$p\\_{t}\\left(\\mathbf{x}\\_{\\text{m}}(t) \\middle| \\mathbf{x}\\_{\\mathcal{I}\\_{\\text{obs}}}(t),\\mathbf{x}\\_{\\mathcal{I}\\_{\\text{obs}}}(0) \\right) \\approx p\\_{t}\\left( \\mathbf{x}\\_{\\text{m}}(t) \\middle| \\mathbf{x}\\_{\\mathcal{I}\\_{\\text{obs}}}(t) \\right).$$ \nFor small $t$,  $\\mathbf{x}\\_{\\mathcal{I}\\_{\\text{obs}}}(t)$ is almost the same as $\\mathbf{x}\\_{\\mathcal{I}\\_{\\text{obs}}}(0)$ so the approximation holds. For large $t$,  $\\mathbf{x}\\_{\\mathcal{I}\\_{\\text{obs}}}(0)$ becomes further away from $\\mathbf{x}\\_{\\text{m}}(t)$ in the Markov chain, and thus have smaller impact on $\\mathbf{x}\\_{\\text{m}}(t)$. Moreover, the approximation error for large $t$ have less effect on the final sample, since it is used early in the sampling process. \nAnother simplification (in Eq.9) is \n$$\\nabla\\_{\\mathbf{x}\\_{\\text{m}}}{\\log{\\mathbb{E}\\_{p_{t}(\\mathbf{x}\\_{\\mathcal{I}\\_{\\text{obs}}}{(t)}|\\mathbf{x}\\_{\\mathcal{I}\\_{\\text{obs}}}{(0)})}\\left\\lbrack p\\_{t}\\left( \\mathbf{x}\\_{\\text{m}}(t) \\middle| \\mathbf{x}\\_{\\mathcal{I}\\_{\\text{obs}}}(t) \\right) \\right\\rbrack}} \\approx \\nabla\\_{\\mathbf{x}\\_{\\text{m}}}{\\log{p\\_{t}\\left( \\mathbf{x}\\_{\\text{m}}(t) \\middle| \\mathbf{x}\\_{\\mathcal{I}\\_{\\text{obs}}}(t) \\right)}}.$$ \nThis is because the $\\mathbf{x}\\_{\\mathcal{I}\\_{\\text{obs}}}(t)$ is a random sample from $p\\_{t}\\left(\\mathbf{x}\\_{\\mathcal{I}\\_{\\text{obs}}}(t) \\middle| \\mathbf{x}\\_{\\mathcal{I}\\_{\\text{obs}}}(0) \\right)$ in practice, so that we can simply this expectation term. We will reclarify the simplification according to your suggestion.\n\n**Q4: The experiments in the paper are insufficient, especially the ablation experiments, and the model comparison should be compared with the underlying methods cited in the Multimodal Fusion and Prediction module to demonstrate the effectiveness of the diffusion module.**\n\nA4: In fact, we had compared with the underlying method cited in the Multimodal Fusion and Prediction module in the Ablation study. As shown in Table 3, the method w/o $s\\_k$ and $\\mathcal{D}\\_k$ denotes this underlying method, i.e., no modal recovery has been performed. Compared to the method w/ $s\\_k$ the  effectiveness of diffusion module is proven.\n\n**Q5: The effectiveness of the semantic disambiguation part of diffusion is not evaluated, and the improvement of the recognition performance of Incomplete Multimodality cannot be assessed.**\n\nA5: For the question, you might miss our supplementary material. We had reported the quantitative results of the ablation experiments for semantic disambiguation (i.e., the conditional recovering) in Section 4 in the supplementary materials, and the results demonstrate the improvement of the recognition performance of Incomplete Multimodality.\n\n**Q6: It is not clear whether the improvement comes from the semantic information provided by other modalities during diffusion or rather from the extraction of better intra-modal information features.** \n\nA6: The improvement comes from two points:1. modality recovery (please see the ablation study in the Table3); 2. the semantic information provided by available modalities during diffusion (please see Section 4 in supplementary material).\n\n**Q7: Some closely related surveys and reviews on MER are missing, such as \"Multimodal machine learning: A survey and taxonomy\", and \"Emotion Recognition From Multiple Modalities: Fundamentals and methodologies\".**\n\nA7: Thanks for your suggestion. we will cite some closely related surveys and reviews on MER such as [a][b] you mentioned in the revision.\n\n[a] Baltrušaitis, Tadas, Chaitanya Ahuja, and Louis-Philippe Morency. \"Multimodal machine learning: A survey and taxonomy.\" IEEE TPAMI, 2018.\n\n[b] Zhao, Sicheng, et al. \"Emotion recognition from multiple modalities: Fundamentals and methodologies.\" IEEE Signal Processing Magazine, 2021.\n\n**Q8: Besides the above weakness, I also have the following questions: Is it reasonable and valid for solving reverse-time SDE with some approximate scores? Does it limit the effect of the model?**\n\nA8: The reasonable and valid for solving reverse-time SDE with some approximate scores have been well-proven by Song’s work, such as [c]. The approximation might limit the effect of the model to some degree, the exact density estimation on diffusion models is an open issue and beyond the scope of this work.\n\n[c] Song, Yang. Learning to Generate Data by Estimating Gradients of the Data Distribution. Ph.D thesis, Stanford University, 2022.\n"}, {"review_id": "DihJe6YTTT", "reviewer": "Reviewer_EQy6", "summary": "The authors recover missing modalities in multimodal emotion recognition by diffusion approaches. The test on two emotion databases and report SOTA results.", "questions": "Why do you think that this could generalise in real world applications rather than in quite self-similar data settings?", "limitations": "Affective Computing is at present heavily debated in terms of ELSI factors to the point where the EU considers its massive limitation in practice. This could be reflected by some thoughts.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The usage of diffusion approaches is \"trendy\" and the recovery of missing data in MER is undercatered for, so the paper is particularly timely from a topic point of view.", "weaknesses": "From the abstract, it appears unclear, which modalities are involved, and whether these should be reconstructed for improved recognition (which seems to make limited sense, if the modality is actually missing) or for the sake of sheer synthesis. Even when reading the paper it is hard to capture what the modalities are, quickly, and ultimately, it seems sheer audiovisual (not including biosignals or symbolic information such as text), so the authors should rename to audiovisual. Also, reconstructions should have been visualised in the paper. A proper quantitative evaluation of results is absent, such as by effect sizes or proper significance testing including Bonferroni adjustment. The authors report Acc instead of the field's typical and more appropriate measure UAR. They fail to cite substantial key literature. The choice of data appears (too) homogeneous - better would be to use more different databases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Thanks for recognizing our innovation and promising results, but you might have some misunderstanding. Below, we respond to each Question (Q) with an Answer (A).\n\n**Q1: From the abstract, it appears unclear, which modalities are involved, and whether these should be reconstructed for improved recognition (which seems to make limited sense, if the modality is actually missing) or for the sake of sheer synthesis. Even when reading the paper it is hard to capture what the modalities are, quickly, and ultimately, it seems sheer audiovisual (not including biosignals or symbolic information such as text), so the authors should rename to audiovisual.**\n\nA1: We appreciate your recognizing our motivation and contribution, but there should exist some misunderstandings. In the abstract, we had mentioned some modalities for MER in the first sentence, i.e., “Human multimodal emotion recognition...such as language, vision, and acoustic”. In particular, starting from Section 3.2 (e.g., Figure 2), we point out that we have considered the three heterogeneous modalities: language, vision, and acoustic. Therefore, it is not sheer audiovisual.\nFor the question of whether these should be reconstructed for improved recognition, many prior works had indicated the effectiveness, more analysis is given in [a].\n\n[a] \"GCNet: graph completion network for incomplete multimodal learning in conversation.\" IEEE TPAMI, 2023.\n\n**Q2: Also, reconstructions should have been visualised in the paper.**\n\nA2: In fact, Figure 3 and Figure 4 had provided the distribution visualization of reconstruction results. Please note the reconstruction is performed in a low-dimensional embedding space.\n\n**Q3: The choice of data appears (too) homogeneous - better would be to use more different databases. The authors report Acc instead of the field's typical and more appropriate measure UAR.**\n\nA3: Thanks for your suggestion. The MOSI and MOSEI datasets belong to the English dataset, but the MOSI focuses on movie reviews and MOSEI contains more than  250 topics. For your concern, we conduct extra experiments on another Chinese MER dataset CH-SIMS [b]. In addition, we add the UAR measure you mentioned for comparison. The experimental results are shown in the following table. Obviously, our proposed IMDer consistently achieves better results than the previous best methods: MMIN and GCNet under random missing protocol (MR means missing rate).\n|MR|MMIN|GCNet|Ours|\n|:----|:----|:----|:----|\n| |ACC$_2$/F1/ACC$_5$/UAR$_5$|ACC$_2$/F1/ACC$_5$/UAR$_5$|ACC$_2$/F1/ACC$_5$/UAR$_5$|\n|0.0|74.9/75.0/45.3/42.3|**76.7/76.8**/50.1/46.2|76.3/76.4/**50.7/**47.4****|\n|0.1|73.4/73.2/44.7/42.7|75.2/75.3/48.0/42.9|**75.4/75.5/50.1/46.1**|\n|0.2|71.9/71.5/44.1/41.1|73.7/73.6/45.1/42.5|**74.7/74.5/47.3/45.6**|\n|0.3|70.5/70.0/40.3/33.7|72.3/72.5/43.2/40.3|**74.2/73.6/45.6/45.6**|\n|0.4|69.6/68.9/39.0/32.1|71.1/71.2/41.0/40.5|**73.7/73.3/44.6/45.2**|\n|0.5|67.1/67.1/37.0/29.2|69.7/69.4/40.5/38.5|**72.7/72.3/43.3/45.3**|\n|0.6|66.7/65.6/37.2/28.9|68.9/68.9/39.8/36.5|**71.3/69.8/42.4/43.5**|\n|0.7|64.3/63.8/35.1/28.3|67.8/67.8/35.9/34.5|**69.8/69.6/42.2/38.9**|\n|Average|69.8/69.4/40.3/34.8|71.9/71.9/43.0/40.2|**73.5/73.1/45.8/44.7**|\n\n [b] \"Ch-sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality.\" ACL. 2020\n\n**Q4: A proper quantitative evaluation of results is absent, such as by effect sizes or proper significance testing including Bonferroni adjustment.**\n\nA4: Thanks for your suggestion. We consider the Cohen's d effect size to conduct new experiments on CH-SIMS dataset. The experimental results are shown in the following table, which proves that the performance improvement of our method is not minor. Especially in the 5-emotion class case, the performance improvement of our method is significant due to the fact that all effect sizes are greater than or equal to 0.8 according to prior rules [c].\n| Methods       |Cohen's d Effect sizes on ACC$_2$/F1/ACC$_5$/UAR$_5$  |\n|---------------|--------------------|\n| Ours vs. GCNet | 0.6 / 0.4 / 0.8 / 1.5|\n| Ours vs. MMIN  | 1.3 / 1.2 / 1.6 / 2.2|\n\n[c] Sawilowsky, Shlomo S. \"New effect size rules of thumb.\" Journal of modern applied statistical methods 8.2 (2009): 26.\n\n**Q5: They fail to cite substantial key literature.**\n\nA5: Thanks for your suggestion.  We cannot understand what the substantial key literature refers to due to no any prompt. Of course, we may cite more literature related to multi-modal learning and emotion recognition, such as [d][e] in the revision.\n\n[d] \"Multimodal machine learning: A survey and taxonomy.\" IEEE TPAMI, 2018.\n\n[e] \"Emotion recognition from multiple modalities: Fundamentals and methodologies.\" IEEE Signal Processing Magazine, 2021.\n\n**Q6: Why do you think that this could generalise in real world applications rather than in quite self-similar data settings? Affective Computing is at present heavily debated in terms of ELSI factors to the point where the EU considers its massive limitation in practice. This could be reflected by some thoughts.**\n\nA6: Our method may be generalized into more real-world applications. First, all datasets we used come from real-world and not synthetic data, and the settings are standard. Second, the incomplete MER problem has been extensively studied in recent years, and recovering missing data could provide some extra cues for performance gains in real applications, more analysis is given in [a]. Third, to reduce the difficulty of recovery, we conduct modality recovery in low-dimensional feature distribution space, and pay more attention to the semantic consistency while recovering the modality. \nFor your concern about the topic of Affective Computing, we think it is a rapidly developing open topic and increasing attention has been paid by researchers especially in recent years, please see those works in the Journal of IEEE Trans on Affective Computing. And, we believe the limitations could be gradually overcome in future research."}]

Author rebuttal:

Meta-review: This paper presents the In-complete Multimodality-Diffused emotion recognition method (IMDer) for multimodal emotion recognition under incomplete modalities. The method uses Gaussian noise in the input space and a score-based diffusion model to generate the data from the missing modalities. The experiments are performed on the CMU-MOSI and CMU-MOSEI benchmarks.

The reviewers valued the proposed technical approach and the reported state-of-the-art results in CMU-MOSI and CMU-MOSEI datasets. In particular, Reviewer BscJ considers the proposed IMDer method to be significant, since it could be inspiring for other problems, such as multimodal transformation/fusion. Besides some details on the presentation aspect (e.g. typos, clarifications about figures and related work, missing references, or the need for summarizing the major contributions), the reviewers highlighted some weaknesses on the evaluation. Concretely, three of the reviewers consider the experiments are not sufficient. In particular, Reviewer tA7F considers the two datasets used in the evaluation (CMU-MOSI and CMU-MOSEI) are very similar, and suggests the need for evaluation in more datasets. Authors reported additional experiments with the CH-SIMS dataset (a chinese multimodal sentiment analysis dataset) in their feedback. Their results on the CH-SIMS improve the best previous results reported on CH-SIMS. 

This paper has been further discussed with the reviewers, AC, and SAC. The proposed method and formulation are interesting and solid. However, there are some important weaknesses that need to be addressed in the final version of the paper. Concretely,

1. More details on the datasets and the specific emotion recognition problem.

Authors should provide more details on the datasets used in their experiments (e.g. CMU-MOSEI and CMU-MOSI are collections of videos from youtube (talking heads) of people providing opinions, that have different human annotations, including valence (the emotional dimension that ranges from negative to positive) and emotion categories (these are available just for CMU-MOSEI)). Also, emotion recognition is a very broad topic. Authors should be more specific about the problem addressed in the paper (e.g. emotion signalling in videos from media). 

2. Results on CH-SIMS

Authors should include the new results with CH-SIMS (which made some of the reviewers raise their scores and solved the concern on the homogeneity of the datasets used to evaluate the method). Also, authors should provide the technical details for the tests performed on CH-SIMS (for example, in the experiments on CMU-MOSEI and CMU-MOSI the authors used BERT to encode the text features, but authors did not specify how the text features are obtained for CH-SIMS).

3. More extended discussions on the limitations of the method.

The only limitations discussed in the paper are about the method, but nothing is said about the limitations of the method for emotion recognition use cases. For instance: (1) this method, trained with the datasets used in the paper, will probably not work to recognize emotion signalling in other scenarios (such as human-robot interaction), because emotion signalling might change depending on the context, culture, etc.; or (2) after training the method with CMU-MOSEI / CMU-MOSI, the resulting model will not perform well to perceive experienced emotion, since emotion experience and emotion signalling might differ.

4. Discussion on the ethical implications.

Since emotion recognition is a sensitive topic (as mentioned by Reviewer EQy6), a thoughtful discussion on the ethical implications of this work needs to be included in the paper. For guidelines on how to write the ethical implications discussion authors can check this document:

https://acii-conf.net/2023/wp-content/uploads/2023/03/instructions-ethical-statement.pdf

In particular, please check the following two sections: “Issues related to potential negative societal impact“ and “Issues related to limits of generalizability”.
Decision: Accept (poster)