SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Given an input example (aka sequence data), we can extract the shallow features,  $\mathcal{X} = \{\mathbf{x}_k\}, \mathbf{x}_k \in \mathbb{R}^{L \times D}$ ,  $L$  and  $D$  indicate the sequence length and the feature dimensionality. In the incomplete multimodal case, some modalities are missing either fixedly or randomly by guaranteeing at least one modality is available in  $\mathcal{X}$ . For the three modalities mentioned above, a total of seven missing combinational cases are included, which are reported in Tab. 1. For a convenient statement but without loss of generality, below we recover the missed acoustic modality  $\mathbf{x}_a$ , as shown in Fig. 2.
3.4 Missing Modality Diffused Network
We consider a score network  $s_{\mathrm{m}}$  to model the distribution of missing modality  $\mathbf{m} \in \mathcal{I}_{\mathrm{miss}}$  by perturbing  $\mathbf{x}_{\mathrm{m}}$  with a common SDE:  $\mathrm{dx} = \sigma^t\mathrm{dw}$ ,  $t \in [0,1]$ , like score-based diffusion [24], the corresponding reverse-time SDE can be derived from Eq. (2) and Eq. (8) as:
$$
\mathrm {d} \mathbf {x} _ {\mathrm {m}} = - \sigma^ {2 t} s _ {\mathrm {m}} \left(\mathbf {x} _ {\mathrm {m}}, \mathbf {x} _ {\mathcal {I} _ {\mathrm {o b s}}}, t; \theta_ {\mathrm {m}}\right) \mathrm {d} t + \sigma^ {t} \mathrm {d} \bar {\mathbf {w}}, \tag {14}
$$
where  $\mathbf{x}_{\mathcal{I}_{\mathrm{obs}}}$  is a condition combining all available modalities. When the score network is well-trained based on Eq. (13), we can generate missing data via Eq. (14). Taking the example that the language modality  $\mathbf{x}_l$  and vision modality  $\mathbf{x}_v$  are available, the acoustic modality  $\mathbf{x}_a$  is missing, we build a score network  $s_a$  conditioned on  $\mathbf{x}_l$  and  $\mathbf{x}_v$  to recover  $\mathbf{x}_a$  with the following reverse-time SDE:
$$
\mathrm {d} \mathbf {x} _ {a} = - \sigma^ {2 t} s _ {a} \left(\mathbf {x} _ {a}, \left[ \mathbf {x} _ {l}; \mathbf {x} _ {v} \right], t; \theta_ {a}\right) \mathrm {d} t + \sigma^ {t} \mathrm {d} \bar {\mathbf {w}}, \tag {15}
$$
where  $[\mathbf{x}_l;\mathbf{x}_v]$  denotes a fused feature that is combined of  $\mathbf{x}_l$  and  $\mathbf{x}_v$  by a feature concatenation operation and a 1D convolution layer with a kernel size of 1. Note that the above feature fusion is not required when only one modality is available. To recover the missing modalities using the above equation, we need to solve the reverse-time SDE with numerical methods. Here, we use a general-purpose Euler-Maruyama numerical method that is based on a simple discretization to the SDE [24]. When applied to Eq. (15), we can obtain the following iteration rule
$$
\mathbf {x} _ {a} (t - \Delta t) = \mathbf {x} _ {a} (t) + \sigma^ {2 t} s _ {a} \left(\mathbf {x} _ {a} (t), \left[ \mathbf {x} _ {l}; \mathbf {x} _ {v} \right] (t), t; \theta_ {a}\right) \Delta t + \sigma^ {t} \sqrt {\Delta t} \epsilon_ {t}, \tag {16}
$$
where  $\Delta t$  is a discrete time step size and  $\epsilon_t \sim \mathcal{N}(\mathbf{0},\mathbf{I})$ . Given sufficient iteration steps, we can obtain recovered data  $\widetilde{\mathbf{x}}_a$ . Subsequently,  $\widetilde{\mathbf{x}}_a$  is fed into the reconstruction module to obtain the refined acoustic modality:  $\hat{\mathbf{x}}_a = \mathcal{D}_a(\widetilde{\mathbf{x}}_a)$ , where  $\mathcal{D}_a$  denotes the reconstruction module of acoustic modality. For any missing patterns, the set of recovered features can be denoted as  $\hat{\mathcal{X}}_{\mathrm{miss}} = \{\widehat{\mathbf{x}}_k|k\in \mathcal{I}_{\mathrm{miss}}\}$ , thus the reconstruction loss  $\mathcal{L}_{\mathrm{rec}}$  is denoted as:
$$
\mathcal {L} _ {\mathrm {r e c}} = \sum_ {k \in \mathcal {I} _ {\mathrm {m i s s}}} \| \widehat {\mathbf {x}} _ {k} - \mathbf {x} _ {k} \| _ {2} ^ {2}. \tag {17}
$$
Further, to optimize the score networks, we can obtain the score matching objective  $\mathcal{L}_{\mathrm{score}}$  under any missing patterns according to Eq. (13):
$$
\mathcal {L} _ {\text {s c o r e}} = \sum_ {k \in \mathcal {I} _ {\text {m i s s}}} \mathbb {E} _ {\mathbf {x} _ {k}, \mathbf {x} _ {\mathcal {I} _ {\text {o b s}}}, \mathbf {z} \sim \mathcal {N} (\mathbf {0}, \mathbf {I}), t \sim \mathcal {U} (0, T)} [ \| \sqrt {\lambda (t)} s _ {k} (\mathbf {x} _ {k} (t), \mathbf {x} _ {\mathcal {I} _ {\text {o b s}}} (t), t; \theta_ {k}) + \mathbf {z} \| _ {2} ^ {2} ]. \tag {18}
$$
By combining  $\mathcal{L}_{\mathrm{rec}}$  and  $\mathcal{L}_{\mathrm{score}}$ , the loss of missing modality diffused network is  $\mathcal{L}_{\mathrm{mmd}} = \mathcal{L}_{\mathrm{rec}} + \mathcal{L}_{\mathrm{score}}$ .
Conditioning mechanism. Besides the above, another vital question is how to embed the available modalities (i.e., conditions) into the sample generation. Inspired by cross-modal attention mechanism [27], we exploit this mechanism to fuse intermediate representation of  $\mathbf{x}_k(t), k \in \mathcal{I}_{\mathrm{miss}}$  and  $\mathbf{x}_{\mathcal{I}_{\mathrm{obs}}}(t)$  in  $s_k$  to achieve condition embedding with a flexible manner. In particular, we assume that a certain intermediate representation of  $\mathbf{x}_k(t)$  and  $\mathbf{x}_{\mathcal{I}_{\mathrm{obs}}}(t)$  in  $s_k$  is  $\mathbf{x}_k^{\mathrm{ir}}(t)$  and  $\mathbf{x}_{\mathcal{I}_{\mathrm{obs}}}^{\mathrm{ir}}(t)$ , where  $\mathbf{x}_k^{\mathrm{ir}}(t)$  and  $\mathbf{x}_{\mathcal{I}_{\mathrm{obs}}}^{\mathrm{ir}}(t)$  have same tensor size. The cross-modal attention mechanism can be defined as:
$$
\mathbf {x} _ {\text {c o n d}} ^ {\mathrm {i r}} (t) = \operatorname {s o f t m a x} \left(\frac {\mathbf {Q K} ^ {\top}}{\sqrt {d}}\right) \mathbf {V}, \tag {19}
$$
where  $\mathbf{x}_{\mathrm{cond}}^{\mathrm{ir}}(t)$  denotes the condition embedded representation,  $d$  denotes dimensionality of intermediate representation.  $\mathbf{Q} = \mathbf{x}_k^{\mathrm{ir}}(t)\mathbf{W}_{\mathbf{Q}},\mathbf{K} = \mathbf{x}_{\mathcal{I}_{\mathrm{obs}}}^{\mathrm{ir}}(t)\mathbf{W}_{\mathbf{K}}$  , and  $\mathbf{V} = \mathbf{x}_{\mathcal{I}_{\mathrm{obs}}}^{\mathrm{ir}}(t)\mathbf{W}_{\mathbf{V}}$  , where  $\mathbf{W}_{\mathbf{Q}},\mathbf{W}_{\mathbf{K}}$  and  $\mathbf{W}_{\mathbf{V}}$  are the learnable parameters. Since we consider a UNet-style architecture to build the score network, we apply Eq. (19) for condition embedding at each different size of intermediate representation. More details of the score network are elaborated in the supplementary file.
3.5 Multimodal Fusion and Prediction
The recovered data  $\widehat{\mathcal{X}}_{\mathrm{miss}} = \{\widehat{\mathbf{x}}_k|k\in \mathcal{I}_{\mathrm{miss}}\}$  and the available data  $\mathcal{X}_{\mathrm{obs}} = \{\mathbf{x}_k|k\in \mathcal{I}_{\mathrm{obs}}\}$  are combined as the complete multimodal data for multimodal fusion and prediction of emotions. We employ multimodal transformers  $\mathcal{T}_k$  [27] to fuse complete multimodal data  $\widehat{\mathcal{X}}_{\mathrm{miss}}\cup \mathcal{X}_{\mathrm{obs}}$  , and the fused feature is used to predict emotions by fully connected layers. We integrate the above losses to reach the full optimization objective:
$$
\mathcal {L} _ {\text {t o t a l}} = \mathcal {L} _ {\text {t a s k}} + \beta \mathcal {L} _ {\text {m m d}}, \tag {20}
$$
where  $\mathcal{L}_{\mathrm{task}}$  is the task loss defined as mean absolute error,  $\beta$  controls the importance of different loss functions. The entire optimization is implemented in an end-to-end manner, and the concrete training about the configuration of incomplete modalities could be found in the experiment part.
Datasets. We consider two standard MER datasets to conduct experiments, including CMU-MOSI [32] and CMU-MOSEI [33]. CMU-MOSI consists of 2199 monologue video clips. Where 1284, 229, and 686 samples are used as training, validation, and testing set. CMU-MOSEI contains 22856 samples of movie review video clips. Where 16326 samples are used for training, the remaining 1871 and 4659 samples are used for validation and testing. On the two datasets, we extract the language features via pre-trained BERT model [9] and obtain a 768-dimensional hidden state as the word embedding. For vision modality, each video frame was encoded via Facet [8] to represent the presence of the total 35 facial action units. The acoustic modality was processed by COVAREP [5] to obtain the 74-dimensional features.
Evaluation metrics. CMU-MOSI and CMU-MOSEI are collections of videos from YouTube of people providing opinions, that have different human annotations, including human emotion score and emotion categories (only for CMU-MOSEI). Our work focuses on human emotion score. Each sample in two datasets was labeled with a human emotion score that ranges from -3 to 3, including highly negative, negative, weakly negative, neutral, weakly positive, positive, and highly positive. To make a comprehensive comparison, we evaluate the performance using the following metrics: 7-class accuracy  $(\mathrm{ACC}_7)$ , binary accuracy  $(\mathrm{ACC}_2)$ , and F1 score.
Implementation details. Following [20, 35, 14], we investigate the performance of different methods under two commonly-used protocols: (1) fixed missing protocol. (2) random missing protocol. For the fixed missing protocol, we consistently discard one modality (i.e.,  $\{l,v\}$ ,  $\{l,a\}$ ,  $\{v,a\}$ ) or two modalities (i.e.,  $\{l\}$ ,  $\{v\}$ ,  $\{a\}$ ). For the random missing protocol, the missing patterns are randomized for each sample (i.e., one or two modalities may be missing for each sample). Here, we use the missing rate (MR) to measure the overall missingness of the dataset. The MR is defined as  $\mathrm{MR} = 1 - \frac{\sum_{i=1}^{N} m_i}{N \times M}$ , where  $m_i$  denotes the number of available modalities for  $i^{th}$  sample,  $N$  denotes the total number of samples, and  $M$  indicates the number of modalities. We also ensure that at least one modality is available for each sample, so  $m_i \geq 1$  and  $\mathrm{MR} \leq \frac{M-1}{M}$ . For three modalities, we choose the MR from  $[0.0, 0.1, \dots, 0.7]$ , where 0.7 is an approximation of  $\frac{M-1}{M}$  with the same meaning. We keep the same MR during training, validation, and testing phases, consistent with previous work [14]. The optimal setting for  $\beta$  is set to 0.1 via the performance on the validation set. We run each experiment five times and report the average values on the testing set.
Table 1: MER accuracy comparison under fixed missing protocol. The values reported in each cell denote  $\mathrm{ACC}_2 / \mathrm{F}1 / \mathrm{ACC}_7$ . Bold is the best.
Datasets	Available	DCCA [2]	DCCAE [29]	MCTN [20]	MMIN [35]	GCNet [14]	IMDer (Ours)
MOSI	{l}	73.6 / 73.8 / 30.2	76.4 / 76.5 / 28.3	79.1 / 79.2 / 41.0	83.8 / 83.8 / 41.6	83.7 / 83.6 / 42.3	84.8 / 84.7 / 44.8
{v}	47.7 / 41.5 / 16.6	52.6 / 51.1 / 17.1	55.0 / 54.4 / 16.3	57.0 / 54.0 / 15.5	56.1 / 55.7 / 16.9	61.3 / 60.8 / 22.2
{a}	50.5 / 46.1 / 16.3	48.8 / 42.1 / 16.9	56.1 / 54.5 / 16.5	55.3 / 51.5 / 15.5	56.1 / 54.5 / 16.6	62.0 / 62.2 / 22.0
{l, v}	74.9 / 75.0 / 30.3	76.7 / 76.8 / 30.0	81.1 / 81.2 / 42.1	83.8 / 83.9 / 42.0	84.3 / 84.2 / 43.4	85.5 / 85.4 / 45.3
{l, a}	74.7 / 74.8 / 29.7	77.0 / 77.0 / 30.2	81.0 / 81.0 / 43.2	84.0 / 84.0 / 42.3	84.5 / 84.4 / 43.4	85.4 / 85.3 / 45.0
{v, a}	50.8 / 46.4 / 16.6	54.0 / 52.5 / 17.4	57.5 / 57.4 / 16.8	60.4 / 58.5 / 19.5	62.0 / 61.9 / 17.2	63.6 / 63.4 / 23.8
{l, v, a}	75.3 / 75.4 / 30.5	77.3 / 77.4 / 31.2	81.4 / 81.5 / 43.4	84.6 / 84.4 / 44.8	85.2 / 85.1 / 44.9	85.7 / 85.6 / 45.3
Average	63.9 / 61.9 / 20.0	66.1 / 64.8 / 24.4	70.2 / 69.9 / 31.3	72.7 / 71.4 / 31.6	73.1 / 72.8 / 32.1	75.5 / 75.3 / 35.5
MOSEI	{l}	78.5 / 78.7 / 46.7	79.7 / 79.5 / 47.0	82.6 / 82.8 / 50.2	82.3 / 82.4 / 51.4	83.0 / 83.2 / 51.2	84.5 / 84.5 / 52.5
{v}	61.9 / 55.7 / 41.3	61.1 / 57.2 / 40.1	62.6 / 57.1 / 41.6	59.3 / 60.0 / 40.7	61.9 / 61.6 / 41.7	63.9 / 63.6 / 42.6
{a}	62.0 / 50.2 / 41.1	61.4 / 53.8 / 40.9	62.7 / 54.5 / 41.4	58.9 / 59.5 / 40.4	60.2 / 60.3 / 41.1	63.8 / 60.6 / 41.7
{l, v}	80.3 / 79.7 / 46.6	80.4 / 80.4 / 47.1	83.2 / 83.2 / 50.4	83.8 / 83.4 / 51.2	84.3 / 84.4 / 51.1	85.0 / 85.0 / 53.1
{l, a}	79.5 / 79.2 / 46.7	80.0 / 80.0 / 47.4	83.5 / 83.3 / 50.7	83.7 / 83.3 / 52.0	84.3 / 84.4 / 51.3	85.1 / 85.1 / 53.1
{v, a}	63.4 / 56.9 / 41.5	62.7 / 59.2 / 41.6	63.7 / 62.7 / 42.1	63.5 / 61.9 / 41.8	64.1 / 57.2 / 42.0	64.9 / 63.5 / 42.8
{l, v, a}	80.7 / 80.9 / 47.7	81.2 / 81.2 / 48.2	84.2 / 84.2 / 51.2	84.3 / 84.2 / 52.4	85.2 / 85.1 / 51.5	85.1 / 85.1 / 53.4
Average	72.3 / 68.8 / 44.5	72.4 / 70.2 / 44.6	74.6 / 72.5 / 46.8	73.7 / 73.5 / 47.1	74.7 / 73.7 / 47.1	76.0 / 75.3 / 48.5
4.2 Comparison with the state-of-the-arts
We compare IMDer with the current state-of-the-art incomplete multimodal emotion recognition methods based on modality recovery mechanism, including MCTN [20], MMIN [35], GCNet [14]. For a comprehensive comparison, we consider two deep learning-based non-recovery methods with
canonical correlation maximization DCCA [2] and DCCAE [29] in incomplete multimodal learning to conduct comparisons. Below, we report the quantitative and qualitative experimental results.
Table 2: MER accuracy comparison under random missing protocol. The values reported in each cell denote  $\mathrm{ACC}_2 / \mathrm{F1} / \mathrm{ACC}_7$ . MR: Missing Rate. Bold is the best.
Datasets	MR	DCCA [2]	DCCAE [29]	MCTN [20]	MMIN [35]	GCNet [14]	IMDer (Ours)
MOSI	0.0	75.3 / 75.4 / 30.5	77.3 / 77.4 / 31.2	81.4 / 81.5 / 43.4	84.6 / 84.4 / 44.8	85.2 / 85.1 / 44.9	85.7 / 85.6 / 45.3
0.1	72.1 / 72.2 / 28.0	74.5 / 74.7 / 28.1	78.4 / 78.5 / 39.8	81.8 / 81.8 / 41.2	82.3 / 82.3 / 42.1	84.9 / 84.8 / 44.8
0.2	69.3 / 69.1 / 26.8	71.8 / 71.9 / 27.6	75.6 / 75.7 / 38.5	79.0 / 79.1 / 38.9	79.4 / 79.5 / 40.0	83.5 / 83.4 / 44.3
0.3	65.4 / 65.2 / 25.7	67.0 / 66.7 / 25.8	71.3 / 71.2 / 35.5	76.1 / 76.2 / 36.9	77.2 / 77.2 / 38.2	81.2 / 81.0 / 42.5
0.4	62.8 / 62.0 / 24.2	63.6 / 62.8 / 24.2	68.0 / 67.6 / 32.9	71.7 / 71.6 / 34.9	74.3 / 74.4 / 36.6	78.6 / 78.5 / 39.7
0.5	60.9 / 59.9 / 21.6	62.0 / 61.3 / 23.0	65.4 / 64.8 / 31.2	67.2 / 66.5 / 32.2	70.0 / 69.8 / 33.9	76.2 / 75.9 / 37.9
0.6	58.6 / 57.3 / 21.2	59.6 / 58.5 / 20.9	63.8 / 62.5 / 29.7	64.9 / 64.0 / 29.1	67.7 / 66.7 / 29.8	74.7 / 74.0 / 35.8
0.7	57.4 / 56.0 / 20.4	58.1 / 57.4 / 20.6	61.2 / 59.0 / 27.5	62.8 / 61.0 / 28.4	65.7 / 65.4 / 28.1	71.9 / 71.2 / 33.4
Average	65.2 / 64.6 / 24.8	66.7 / 66.3 / 25.2	70.6 / 70.1 / 34.8	73.5 / 73.1 / 35.8	75.2 / 75.1 / 36.7	79.6 / 79.3 / 40.5
MOSEI	0.0	80.7 / 80.9 / 47.7	81.2 / 81.2 / 48.2	84.2 / 84.2 / 51.2	84.3 / 84.2 / 52.4	85.2 / 85.1 / 51.5	85.1 / 85.1 / 53.4
0.1	77.4 / 77.3 / 46.2	78.4 / 78.3 / 46.9	81.8 / 81.6 / 49.8	81.9 / 81.3 / 50.6	82.3 / 82.1 / 51.2	84.8 / 84.6 / 53.1
0.2	73.8 / 74.0 / 45.1	75.5 / 75.4 / 46.3	79.0 / 78.7 / 48.6	79.8 / 78.8 / 49.6	80.3 / 79.9 / 50.2	82.7 / 82.4 / 52.0
0.3	71.1 / 71.2 / 43.6	72.3 / 72.2 / 45.6	76.9 / 76.2 / 47.4	77.2 / 75.5 / 48.1	77.5 / 76.8 / 49.2	81.3 / 80.7 / 51.3
0.4	69.5 / 69.4 / 43.1	70.3 / 70.0 / 44.0	74.3 / 74.1 / 45.6	75.2 / 72.6 / 47.5	76.0 / 74.9 / 48.0	79.3 / 78.1 / 50.0
0.5	67.5 / 65.4 / 42.5	69.2 / 66.4 / 43.3	73.6 / 72.6 / 45.1	73.9 / 70.7 / 46.7	74.9 / 73.2 / 46.7	79.0 / 77.4 / 49.2
0.6	66.2 / 63.1 / 42.4	67.6 / 63.2 / 42.9	73.2 / 71.1 / 43.8	73.2 / 70.3 / 45.6	74.1 / 72.1 / 45.1	78.0 / 75.5 / 48.5
0.7	65.6 / 61.0 / 42.1	66.6 / 62.6 / 42.5	72.7 / 70.5 / 43.6	73.1 / 69.5 / 44.8	73.2 / 70.4 / 44.5	77.3 / 74.6 / 47.6
Average	71.5 / 70.3 / 44.1	72.6 / 71.2 / 45.0	77.0 / 76.1 / 46.9	77.3 / 75.4 / 48.2	77.9 / 76.8 / 48.3	80.9 / 79.8 / 50.6
Quantitative results. Tab. 1 and Tab. 2 list the quantitative results of fixed missing protocol and random missing protocol on two datasets, respectively. We have the following observations:
1. IMDer achieves the best results on the two MER datasets under both fixed missing protocol and random missing protocol, demonstrating superiority of IMDer. Compared with the non-recovery-based methods [2, 29], IMDer obtains consistent MER performance gains. This can be explained that IMDer can explicitly recover the missing modalities which can provide extra complementary information for MER. Compared with the recovery-based methods [20, 35, 14], the consistent improvements of IMDer indicate the superiority of maintaining distribution consistency between the recovered and the original modalities.
Figure 3: Visualization of recovered modalities. Avail. means available. The distribution of modality recovered by our method is much closer to the ground truth than the compared methods.
2. From the experimental results under fixed missing protocol in Tab. 1, we can observe that when language modality is missing, our proposed IMDer obtains more obvious enhancements. This is reasonable because language modality is crucial for robust MER, i.e., it is highly abstract and semantic and contains more discriminative information [20]. Thus, it is vital to complete the missing language modality for robust MER. Compared with other recovery-based methods, our proposed IMDer explicitly models the distribution space of the language modality and recovers the distributionally and
semantically consistent language features to facilitate robust MER. The comprehensive comparison in Tab. 1 verify the effectiveness of our method.
3. Compared with other MER methods, IMDer suffers from less performance degradation as the modality missing rate increases. This can be verified in Tab. 2 (random missing protocol). On CMU-MOSI, when the missing rate increases from 0.0 to 0.7, the  $\mathrm{ACC}_2$  of the compared recovery-based methods declines  $19.5\% \sim 21.8\%$  while our IMDer merely declines  $13.8\%$ . For the CMU-MOSEI, as the missing rate increases from 0.0 to 0.7,  $\mathrm{ACC}_2$  of other recovery-based methods show obvious decline:  $11.2\% \sim 12\%$ . As a comparison, our proposed IMDer shows  $7.8\%$  accuracy degradation. These comparisons demonstrate IMDer's generalizability under random missing protocol.
Qualitative results. Fig. 3 visualizes the distribution of recovered data and ground truth for different recovery-based methods under the fixed missing protocol. We randomly select 500 samples in the testing set from the CMU-MOSEI dataset. The features of the selected samples are projected into a 2D space by t-SNE [28]. From these results, we can observe that the distribution between the original and recovered modalities estimated by IMDer is much closer than other methods, indicating IMDer effectively mitigates the distribution gap between recovered and the original data.
Table 3: Ablation study of the key components in IMDer under the average random missing protocol.
Datasets	sk	Dk	ACC2	F1	ACC7
CMU-MOSI	✓	✓	79.6	79.3	40.5
✓	×	77.8	77.7	38.4
×	✓	75.8	75.6	36.8
×	×	74.5	74.4	36.0
CMU-MOSEI	✓	✓	80.9	79.8	50.6
✓	×	79.8	78.5	49.4
×	✓	78.0	76.9	48.8
×	×	76.5	76.0	48.5
Figure 4: Visualization of the unconditional and conditional modality recovering. Conditional recovering (bottom row) shows the promising emotion category separability, indicating that using the vanilla available modalities as conditions can reduce the semantic ambiguity.
Quantitative analysis. We evaluate the effects of the key components in IMDer, including score model  $s_k$  and reconstruction module  $\mathcal{D}_k$ . The results are illustrated in Tab. 3, we conclude the conclusions as: 1) Recovering missing modalities by reconstruction module  $\mathcal{D}_k$  or score model  $s_k$  is effective. It is because explicitly recovering the modality can provide meaningful complementary
information. Further, we find that the score model-based modality recovery can provide more performance improvements. This is because the score model can recover data from the distribution of each missing modality, and the distribution between recovered data and original data is more consistent. 2) Combing  $s_k$  with  $\mathcal{D}_k$  brings further benefits, which proves the refinement mechanism provided by  $\mathcal{D}_k$  and the diffusion-based score model  $s_k$  can benefit each other mutually.
Visualization of unconditional/conditional recovering. Fig. 4 visualizes the recovered and the original modality under the unconditional and conditional settings. Specifically, we randomly select 70 samples (ten samples for each class) in the testing set of the CMU-MOSEI dataset. The features of the selected samples are projected into a 2D space by t-SNE [28]. We can observe that unconditional recovering suffer from semantic ambiguity due to the lack of semantic guidance. In contrast, the data recovered by conditional recovering has been clustered and can be distinguished across seven different emotion classes, indicating that conditional recovering can mitigate semantic ambiguity between the recovered and the original modalities.
In this paper, we try to challenge MER under incomplete multimodilities by proposing an Incomplete Multimodality-Diffused emotion recognition (IMDer) method. To recover the missing modalities, IMDer exploits the score-based diffusion model that maps the input random noise into the distribution space of missing modalities and recovers missing data abided by their original distributions. To reduce semantic ambiguity between the missing and the recovered modalities, the available modalities are used as conditions to guide the diffusion-based recovering process. Quantitative and qualitative experiments consistently demonstrate the effectiveness of IMDer.
Limitations. This work also has some limitations. The first limitation is that solving the reverse-time SDE incurs high computational overhead and may not be suitable for scenarios requiring real-time performance, and we will try to mitigate this issue in future work, such as replacing the Euler-Maruyama solvers with ordinary differential equation solvers. In addition, our work may not work to recognize emotion in other scenarios such as human-robot interaction, because emotion signal might change depending on the context, culture, etc.
Ethical implications discussion. Recognizing human emotion from multimodality helps build better human-computer interactions. Albeit the obvious benefits, this technology also introduces some potential negative societal impact. For example, multimodal data contains a large number of faces, facial data will bring potential social privacy and security issues.
This work was supported by the National Natural Science Foundation of China (Grant Nos. 62072244, 62102180), the fundamental research funds for the central universities (Grant No. 30919011232), the Natural Science Foundation of Shandong Province (Grant Nos. ZR2020LZH008, ZR2022LZH003), the Natural Science Foundation of Jiangsu Province (Grant No. BK20210329), Shuangchuang Program of Jiangsu Province (Grant No. JSSCBS20210210), and in part by State Key Laboratory of High-end Server & Storage Technology.
[1] Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326, 1982. 3
[2] Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International conference on machine learning, pages 1247-1255. PMLR, 2013. 3, 7, 8
[3] Qiong Cai, Hao Wang, Zhenmin Li, and Xiao Liu. A survey on multimodal data-driven smart healthcare systems: approaches and applications. IEEE Access, 7:133583-133599, 2019. 1
[4] Zhen Cui, Ling Zhou, Chaoqun Wang, Chunyan Xu, and Jian Yang. Visual micro-pattern propagation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):1267-1286, 2022. 3
[5] Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. Covarep—a collaborative voice analysis repository for speech technologies. In 2014 IEEE international conference on acoustics, speech and signal processing (icassp), pages 960-964. IEEE, 2014. 7
[6] Harold Hotelling. Relations between two sets of variates. Breakthroughs in statistics: methodology and distribution, pages 162-190, 1992. 3