SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuanzhi Wang, Yong Li, Zhen Cui
PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China. {yuanzhiwang, yong.li, zhen.cui}@njust.edu.cn
Human multimodal emotion recognition (MER) aims to perceive and understand human emotions via various heterogeneous modalities, such as language, vision, and acoustic. Compared with unimodality, the complementary information in the multimodilities facilitates robust emotion understanding. Nevertheless, in real-world scenarios, the missing modalities hinder multimodal understanding and result in degraded MER performance. In this paper, we propose an Incomplete Multimodality-Diffused emotion recognition (IMDer) method to mitigate the challenge of MER under incomplete multimodality. To recover the missing modalities, IMDer exploits the score-based diffusion model that maps the input Gaussian noise into the desired distribution space of the missing modalities and recovers missing data abided by their original distributions. Specially, to reduce semantic ambiguity between the missing and the recovered modalities, the available modalities are embedded as the condition to guide and refine the diffusion-based recovering process. In contrast to previous work, the diffusion-based modality recovery mechanism in IMDer allows to simultaneously reach both distribution consistency and semantic disambiguation. Feature visualization of the recovered modalities illustrates the consistent modality-specific distribution and semantic alignment. Besides, quantitative experimental results verify that IMDer obtains state-of-the-art MER accuracy under various missing modality patterns. Codes are released at https://github.com/mdswyz/IMDer.
Benefiting from the intrinsic heterogeneity of multimodal data, various modalities are exploited for multimodal emotion recognition (MER) to understand human behaviors and intents from a collaborative perspective [27, 13, 36]. Recently, MER has become one of the most active research topics of affective computing with many applications, such as healthcare [3, 18] and robotics [11, 12].
Robust MER relies on learning and combining representations from diverse modalities [27]. In previous studies, Zadeh et al. [31] designed a Tensor Fusion Network that takes as input paired modalities to encode the bimodal representation, which are further fused to generate the trimodal representation. Tsai et al. [27] proposed a multimodal transformer to learn the potential adaptation and correlations between modalities. Subsequently, various advanced approaches [17, 15, 13] have explored different variants of multimodal transformers to construct robust MER frameworks.
However, in real-world scenarios, not all modalities are consistently available, e.g., language data may be missing due to speech recognition errors; video data may be inaccessible due to privacy
Corresponding authors: Yong Li, Zhen Cui
Figure 1: (a) shows the convenient modality recovery-based MER method. Typically, a well-crafted encoder-decoder architecture is used to recover the missing modalities. (b) illustrates our proposed Incomplete Multimodality-Diffused emotion recognition (IMDer) method. IMDer exploits the vanilla available modalities as the condition to guide and refine the distribution space of the recovered modality, thus achieves distribution consistency and semantic disambiguation.
and security concerns [14]. Ultimately, these incomplete multimodal data significantly hinder the performance of MER. For MER under incomplete multimodality, one straightforward way is to recover the missing modalities from the available ones. As shown in Fig. 1 (a), convenient modality recovery methods [20, 35, 14, 30] aim to recover the missing modalities via a well-designed encoder-decoder framework and build a mapping between the vanilla available and the missing modalities. Among them, Zhao et al. [35] combined autoencoder with cycle consistency learning for modality recovery. Lian et al. [14] designed a graph completion network that utilized graph neural networks to reconstruct missing parts. However, these previous methods fail to explicitly consider the modality-specific distributions that are highly correlated with each modality's intrinsic discriminability, e.g., an image gives the visual appearance of a happy face via thousands of pixels, whilst the corresponding text describes this emotion with a sentence using discrete words.
In this paper, we aim to challenge MER under incomplete multimodities by proposing an Incomplete Multimodality-Diffused emotion recognition (IMDer) method, as shown in Fig. 1 (b). To recover the missing modalities, IMDer exploits the prevalent score-based diffusion model [24] that maps the input random noise into the distribution space of the missing modalities. In particular, the score-based diffusion model captures the distribution of the missing modalities by perturbing data with the stochastic differential equation (SDE). With sufficient data and model capacity, we are capable of exploiting the well-trained score model to recover the distribution-consistent modality by solving reverse-time SDE (i.e., denoising process) starting from the prior noise distribution.
To reduce the semantic ambiguity between the missing and the corresponding recovered modalities, we use the vanilla available modalities as semantic conditions to guide and refine the recovering process. The mere information embedded in the available modalities facilitates IMDer simultaneously reaching both distribution consistency and semantic disambiguation. Finally, the recovered modalities together with available modalities could be jointly fed into a multimodal fusion and prediction network for MER. In summary, the contributions of this work can be concluded as:
- To address the challenge of MER under incomplete multimodalities, we propose the Incomplete Multimodality-Diffused emotion recognition (IMDer) method. IMDer maps input random noise to the distribution space of missing modalities and recovers missing data in accordance with their original distributions.
- To minimize the semantic ambiguity between the missing and recovered modalities, we utilize the available modalities as prior conditions to guide and refine the recovering process. This ensures that the recovered modality is both distribution and semantic-consistent.
- We perform extensive experiments on publicly available MER datasets and achieve superior or comparable results across different missing modality patterns. The feature visualization of the recovered modalities demonstrates consistent distribution and semantic alignment.
2.1 Incomplete Multimodal Learning
Incomplete multimodal learning is an essential research topic in multimodal machine learning, as it can deal with the inevitable modality-missing environments in real-world scenarios. One effective approach is to find a low-dimensional subspace shared by all modalities, in which different modalities
have maximum correlation with each other [6, 2, 29]. However, this approach may ignore the complementarity between heterogeneous modalities, leading to suboptimal results for downstream tasks. Another more effective approach is to explicitly recover the missing modalities based on available ones. Representative methods include zero-based recovery [19], average-based recovery [34], and deep learning-based recovery [20]. Of these, the zero and the average-based recovery methods still cause a considerable gap between the recovered and the original data because they do not utilize any supervision information. In contrast, deep learning-based methods can better estimate missing modalities by leveraging their powerful feature representation capabilities. For example, Tran et al. [26] recovered missing modalities with a cascaded residual autoencoder. Pham et al. [20] and Zhao et al. [35] aimed to learn the joint multimodal representation via the cross-modality recovery strategy with cycle consistency loss for missing modalities issue. Lian et al. [14] utilized graph neural networks [4] to reconstruct missing modalities to improve MER accuracy.
2.2 Score-based Generative Diffusion Model
The score-based generative model is a family of generative models [22-24], which models data distribution through the Stein score function [25, 16]. Given a probability density function  $p(\mathbf{x})$ , the Stein score function (abbreviated as score) is defined as  $s(\mathbf{x}) = \nabla_{\mathbf{x}} \log p(\mathbf{x})$ . As we might guess, the score-based generative model aims to parameterize a score network  $s(\mathbf{x}; \theta)$  with learnable parameters  $\theta$  and train it to estimate  $\nabla_{\mathbf{x}} \log p(\mathbf{x})$ . Unlike likelihood-based models such as normalizing flows [10], score-based models do not have to be normalized and are easier to parameterize.
Perturbing data with a diffusion process. To generate samples with score-based models, we need to consider a diffusion process that corrupts data slowly into random noise. Let  $\{\mathbf{x}(t) \in \mathbb{R}^d\}_{t=0}^T$  be a diffusion process, indexed by the continuous time variable  $t \in [0,T]$ . From the landmark work [24], a diffusion process is built by a stochastic differential equation (SDE), formally,
$$
\mathrm {d} \mathbf {x} = \mathbf {f} (\mathbf {x}, t) \mathrm {d} t + g (t) \mathrm {d} \mathbf {w}, \tag {1}
$$
where  $\mathbf{f}(\cdot, t)$  denotes the drift coefficient of the SDE,  $g(t)$  is called the diffusion coefficient, and  $\mathbf{w}$  represents the standard Brownian motion. Further, the distribution of  $\mathbf{x}(t)$  is denoted as  $p_t(\mathbf{x}(t))$ , such that  $\mathbf{x}(0) \sim p_0$ , and  $\mathbf{x}(T) \sim p_T$ .  $p_0$  is the data distribution, and  $p_T$  is the prior distribution.
Generating samples by reversing the diffusion process. By starting from sample  $\mathbf{x}(T) \sim p_T$  and reversing the diffusion process, we can generate a sample from  $p_0$ . Since the reverse of a diffusion process is also a diffusion process [1], running backwards in time and given by the reverse-time SDE:
$$
\mathrm {d} \mathbf {x} = [ \mathbf {f} (\mathbf {x}, t) - g (t) ^ {2} \nabla_ {\mathbf {x}} \log p _ {t} (\mathbf {x}) ] \mathrm {d} t + g (t) \mathrm {d} \bar {\mathbf {w}}, \tag {2}
$$
where  $\bar{\mathbf{w}}$  is a Brownian motion in the reverse time direction. This reverse-time SDE can be computed once we know the drift and diffusion coefficients of Eq. (1), as well as the score for each  $t\in [0,T]$ .
Estimating score via score matching. From the above, it is known that the core of solving reverse-time SDE is to obtain the score for each time step  $t$ . The score can be estimated by training a time-dependent score network  $s(\mathbf{x}(t), t; \theta)$  on training samples with the score matching [7, 24]:
$$
\mathcal {L} _ {\text {s c o r e}} = \mathbb {E} _ {t \sim \mathcal {U} (0, T)} [ \lambda (t) \mathbb {E} _ {\mathbf {x} (0)} \mathbb {E} _ {\mathbf {x} (t) | \mathbf {x} (0)} [ \| s (\mathbf {x} (t), t; \theta) - \nabla_ {\mathbf {x} (t)} \log p _ {0 t} (\mathbf {x} (t) \mid \mathbf {x} (0)) \| _ {2} ^ {2} ] ], \tag {3}
$$
where  $\mathcal{U}(0,T)$  is a uniform distribution over  $[0,T]$ ,  $p_{0t}(\mathbf{x}(t) \mid \mathbf{x}(0))$  denotes the transition kernel from  $\mathbf{x}(0)$  to  $\mathbf{x}(t)$ , and  $\lambda(t)$  denotes a positive weighting function. Finally, we need to know the transition kernel  $p_{0t}(\mathbf{x}(t) \mid \mathbf{x}(0))$  to minimize Eq. (3) and optimize  $s(\mathbf{x}(t), t; \theta)$ . The transition kernel is always a Gaussian distribution when the drift coefficient of SDE is affine, where the mean and variance can be obtained with common techniques (see section 5.5 in [21]). For example, consider a SDE for score models:  $\mathrm{d}\mathbf{x} = \sigma^t\mathrm{d}\mathbf{w}$ ,  $t \in [0,1]$ , in this case, the corresponding transition kernel is
$$
p _ {0 t} (\mathbf {x} (t) \mid \mathbf {x} (0)) = \mathcal {N} \left(\mathbf {x} (t); \mathbf {x} (0), \frac {1}{2 \log \sigma} \left(\sigma^ {2 t} - 1\right) \mathbf {I}\right). \tag {4}
$$
When we choose the weighting function  $\lambda(t) = \frac{1}{2\log\sigma} (\sigma^{2t} - 1)$ , the final objective  $\mathcal{L}_{\mathrm{score}}$  is
$$
\mathcal {L} _ {\text {s c o r e}} = \mathbb {E} _ {\mathbf {x}, \mathbf {z} \sim \mathcal {N} (\mathbf {0}, \mathbf {I}), t \sim \mathcal {U} (0, T)} [ \| \sqrt {\lambda (t)} s (\mathbf {x} (t), t; \theta) + \mathbf {z} \| _ {2} ^ {2} ], \tag {5}
$$
where  $\mathbf{z}$  is the random noise sampling from  $\mathcal{N}(\mathbf{0},\mathbf{I})$
Let a tuple  $(\mathbf{x}_1, \mathbf{x}_2, \dots, \mathbf{x}_M)$  denotes  $M$  heterogenous modalities of an example, where  $\mathbf{x}_k$  is the input of the  $k$ -th modality. In the complete case, all modalities are observed and directly fused to facilitate MER. However, in many limited scenarios, there may be some modalities that are unavailable and need to be recovered for better fusion. For simplification, we introduce an indicator  $\alpha \in \{0, 1\}$  to denote with  $\alpha_k = 0$  if the  $k$ -th modality is missing, otherwise  $\alpha_k = 1$ . In the incomplete case, thus, the target is to recover those unobserved modalities  $\mathcal{I}_{\mathrm{miss}} = \{k | \alpha_k = 0\}$  for boosting the downstream tasks. It is worth noting that the missing emotion modalities may not be consistent for each example. To complete the missing modality  $k \in \mathcal{I}_{\mathrm{miss}}$ , we could sample a large amount of data from its latent distribution  $p(\mathbf{x}_k)$ , as those advanced generative models do. But the sampling process does not have any semantic guidance, how to generate semantically consistent samples in the sampling process (i.e., avoid semantic ambiguity)? Fortunately, the observed modalities  $\mathcal{I}_{\mathrm{obs}} = \{k | \alpha_k = 1\}$  contain the semantic information we need, and thus can flavor the sampling process.
Our main idea is to recover the missing emotion modalities  $\mathcal{I}_{\mathrm{miss}}$  from their latent distribution spaces on the condition of the available modalities  $\mathcal{I}_{\mathrm{obs}}$ . In other words, the observed emotion modalities  $\mathcal{I}_{\mathrm{obs}}$  are injected into the recovering process as the semantic conditions to constrain the distribution space of the sampling. Formally, we denote the data distribution of one missing modality as  $p(\mathbf{x}_{\mathrm{m}}) (\mathrm{m} \in \mathcal{I}_{\mathrm{miss}})$  and the available ones as  $p(\mathbf{x}_{\mathcal{I}_{\mathrm{obs}}})$ , where  $\mathbf{x}_{\mathcal{I}_{\mathrm{obs}}}$  is abbreviated for brevity. Hereby, the ultimate goal is to sample missing modal data from the conditional distribution  $p(\mathbf{x}_{\mathrm{m}} | \mathbf{x}_{\mathcal{I}_{\mathrm{obs}}})$ . We consider a multi-step diffusion model to build this conditional distribution by perturbing  $\mathbf{x}_{\mathrm{m}}$ , the perturbed conditional transition distribution is  $p_{t}(\mathbf{x}_{\mathrm{m}}(t) | \mathbf{x}_{\mathcal{I}_{\mathrm{obs}}} (0))$  at the  $t$ -th step. Although  $p_{t}(\mathbf{x}_{\mathrm{m}}(t) | \mathbf{x}_{\mathcal{I}_{\mathrm{obs}}} (0))$  is in general intractable, it can be approximated. We have
$$
\begin{array}{l} p _ {t} \left(\mathbf {x} _ {\mathrm {m}} (t) \mid \mathbf {x} _ {\mathcal {I} _ {\mathrm {o b s}}} (0)\right) = \int p _ {t} \left(\mathbf {x} _ {\mathrm {m}} (t) \mid \mathbf {x} _ {\mathcal {I} _ {\mathrm {o b s}}} (t), \mathbf {x} _ {\mathcal {I} _ {\mathrm {o b s}}} (0)\right) p _ {t} \left(\mathbf {x} _ {\mathcal {I} _ {\mathrm {o b s}}} (t) \mid \mathbf {x} _ {\mathcal {I} _ {\mathrm {o b s}}} (0)\right) d \mathbf {x} _ {\mathcal {I} _ {\mathrm {o b s}}} (t) (6) \\ \approx \int p _ {t} (\mathbf {x} _ {\mathrm {m}} (t) | \mathbf {x} _ {\mathcal {I} _ {\mathrm {o b s}}} (t)) p _ {t} (\mathbf {x} _ {\mathcal {I} _ {\mathrm {o b s}}} (t) | \mathbf {x} _ {\mathcal {I} _ {\mathrm {o b s}}} (0)) d \mathbf {x} _ {\mathcal {I} _ {\mathrm {o b s}}} (t) (7) \\ \end{array}
$$
where  $p_t(\mathbf{x}_{\mathcal{I}_{\mathrm{obs}}}(t) | \mathbf{x}_{\mathcal{I}_{\mathrm{obs}}}^{\prime}(0))$  is tractable because it can be derived from the forward diffusion process. In the above approximation, we use the fact that  $\mathbf{x}_{\mathcal{I}_{\mathrm{obs}}}^{\prime}(t)$  is almost the same as  $\mathbf{x}_{\mathcal{I}_{\mathrm{obs}}}^{\prime}(0)$  when  $t$  is small; if  $t$  increases,  $\mathbf{x}_{\mathcal{I}_{\mathrm{obs}}}^{\prime}(0)$  becomes further away from  $\mathbf{x}_{\mathrm{m}}(t)$  in Markov chain so that  $\mathbf{x}_{\mathrm{m}}(t)$  is almost independent of  $\mathbf{x}_{\mathcal{I}_{\mathrm{obs}}}^{\prime}(0)$ . Motivated by the score-based diffusion model [24], we compute the score of conditional transition probability in Eq. (6):
$$
\begin{array}{l} \nabla_ {\mathbf {x} _ {\mathrm {m}}} \log p _ {t} \left(\mathbf {x} _ {\mathrm {m}} (t) \mid \mathbf {x} _ {\mathcal {I} _ {\mathrm {o b s}}} (0)\right) \approx \nabla_ {\mathbf {x} _ {\mathrm {m}}} \log \mathbb {E} _ {p _ {t} \left(\mathbf {x} _ {\mathcal {I} _ {\mathrm {o b s}}} (t) \mid \mathbf {x} _ {\mathcal {I} _ {\mathrm {o b s}}} (0)\right)} \left[ p _ {t} \left(\mathbf {x} _ {\mathrm {m}} (t) \mid \mathbf {x} _ {\mathcal {I} _ {\mathrm {o b s}}} (t)\right) \right] (8) \\ \approx \nabla_ {\mathbf {x} _ {\mathrm {m}}} \log p _ {t} \left(\mathbf {x} _ {\mathrm {m}} (t) \mid \mathbf {x} _ {\mathcal {I} _ {\mathrm {o b s}}} (t)\right) (9) \\ = \nabla_ {\mathbf {x} _ {\mathrm {m}}} \log p _ {t} ([ \mathbf {x} _ {\mathrm {m}} (t); \mathbf {x} _ {\mathcal {I} _ {\mathrm {o b s}}} (t) ]), (10) \\ \end{array}
$$
where in the second step  $\mathbf{x}_{\mathcal{I}_{\mathrm{obs}}}(t)$  is a random sample from  $p_t(\mathbf{x}_{\mathcal{I}_{\mathrm{obs}}}(t)|\mathbf{x}_{\mathcal{I}_{\mathrm{obs}}}(0))$  and note that  $\nabla_{\mathbf{x}_{\mathrm{m}}} \log p_t(\mathbf{x}_{\mathrm{m}}(t)|\mathbf{x}_{\mathcal{I}_{\mathrm{obs}}}(t)) = \nabla_{\mathbf{x}_{\mathrm{m}}} \log p_t([ \mathbf{x}_{\mathrm{m}}(t); \mathbf{x}_{\mathcal{I}_{\mathrm{obs}}}(t) ])$  is held because:
$$
\begin{array}{l} \nabla_ {\mathbf {x} _ {\mathrm {m}}} \log p _ {t} ([ \mathbf {x} _ {\mathrm {m}} (t); \mathbf {x} _ {\mathcal {I} _ {\mathrm {o b s}}} (t) ]) = \nabla_ {\mathbf {x} _ {\mathrm {m}}} \log p _ {t} (\mathbf {x} _ {\mathrm {m}} (t) | \mathbf {x} _ {\mathcal {I} _ {\mathrm {o b s}}} (t)) + \nabla_ {\mathbf {x} _ {\mathrm {m}}} \log p _ {t} (\mathbf {x} _ {\mathcal {I} _ {\mathrm {o b s}}} (t)) (11) \\ = \nabla_ {\mathbf {x} _ {\mathrm {m}}} \log p _ {t} \left(\mathbf {x} _ {\mathrm {m}} (t) \mid \mathbf {x} _ {\mathcal {I} _ {\mathrm {o b s}}} (t)\right), (12) \\ \end{array}
$$
where  $[\mathbf{x}_{\mathrm{m}}(t); \mathbf{x}_{\mathcal{I}_{\mathrm{obs}}}(t)]$  is a vector combination of  $\mathbf{x}_{\mathrm{m}}(t)$  and  $\mathbf{x}_{\mathcal{I}_{\mathrm{obs}}}(t)$  by a well-designed transformation function. This transformation function is called the conditioning mechanism that is described in Sec. 3.4. To infer the gradient,  $\nabla_{\mathbf{x}_{\mathrm{m}}} \log p_t([ \mathbf{x}_{\mathrm{m}}(t); \mathbf{x}_{\mathcal{I}_{\mathrm{obs}}}(t) ])$  could be parameterized a time-dependent conditioning score network  $s_{\mathrm{m}}(\mathbf{x}_{\mathrm{m}}(t), \mathbf{x}_{\mathcal{I}_{\mathrm{obs}}}(t), t; \theta_{\mathrm{m}})$  for modality m. Finally, based on Eq. (5), we derive the conditional score matching objective to be optimized:
$$
\mathcal {L} _ {\text {s c o r e}} ^ {\text {c o n d}} = \mathbb {E} _ {\mathbf {x} _ {\mathrm {m}}, \mathbf {x} _ {\mathcal {I} _ {\text {o b s}}}, \mathbf {z} \sim \mathcal {N} (\mathbf {0}, \mathbf {I}), t \sim \mathcal {U} (0, T)} [ \| \sqrt {\lambda (t)} s _ {\mathrm {m}} (\mathbf {x} _ {\mathrm {m}} (t), \mathbf {x} _ {\mathcal {I} _ {\text {o b s}}} (t), t; \theta_ {\mathrm {m}}) + \mathbf {z} \| _ {2} ^ {2} ]. \tag {13}
$$
Given sufficient data and model capacity, the well-trained  $s_{\mathrm{m}}(\mathbf{x}_{\mathrm{m}}(t), \mathbf{x}_{\mathcal{I}_{\mathrm{obs}}}(t), t; \theta_{\mathrm{m}})$  can be used to solve the reverse-time SDE in Eq. (2) to generate each missing modality  $\widetilde{\mathbf{x}}_{\mathrm{m}}$ , abided by  $p(\mathbf{x}_{\mathrm{m}})$ .
Although the estimated  $\widetilde{\mathbf{x}}_{\mathrm{m}}$  obeys the original distribution, it would deviate from the ground truth when the scatter degree of intra-class samples is large. Thus, we can further refine it with a decoder  $\mathcal{D}$ , i.e.,  $\widehat{\mathbf{x}}_{\mathrm{m}} = \mathcal{D}_{\mathrm{m}}(\widetilde{\mathbf{x}}_{\mathrm{m}})$ . At the training stage, we minimize the reconstruction error between  $\widehat{\mathbf{x}}_{\mathrm{m}}$  and  $\mathbf{x}_{\mathrm{m}}$ . And, we find this refinement step could enhance the performance as shown in Sec. 4.3.
3.2 Overview Framework
Figure 2: The framework of IMDer. Given the input incomplete data (acoustic modality is missed as an example), IMDer encodes shallow feature  $\mathbf{x}_l$  and  $\mathbf{x}_v$  (equal to  $\mathbf{x}_l(0)$  and  $\mathbf{x}_v(0)$ ) via shallow feature extractor (Sec. 3.3). In missing modality diffused network, we sample  $\mathbf{x}_a(T)$  from prior noise distribution, and then denoise  $\mathbf{x}_a(T)$  by solving the reverse-time diffusion with score model  $s_a$  conditioned on  $[\mathbf{x}_l; \mathbf{x}_v]$  to generate samples  $\widetilde{\mathbf{x}}_a(0)$ . Further,  $\widetilde{\mathbf{x}}_a(0)$  is fed into the reconstruction module  $\mathcal{D}_a$  to obtain the final recovered modality  $\widehat{\mathbf{x}}_a$  (Sec. 3.4). Finally,  $\widehat{\mathbf{x}}_a$  is combined with  $\mathbf{x}_l$  and  $\mathbf{x}_v$  as complete data for multimodal fusion and prediction to regress the emotional labels. (Sec. 3.5).
The overview framework is illustrated in Fig. 2. It mainly consists of three parts: shallow feature extractor, missing modality diffused network, and multimodal fusion and prediction. Due to the discrepancy of the original dimensional space between modalities, we first align the dimensionality of each modality by extracting multimodal shallow features to facilitate subsequent modality recovery. Then, to alleviate the distribution gap between recovered modalities and original ones, we design a missing modality diffused network to model the distribution of each missing modality and sample from the modeled distribution space to generate missing data. At the same time, to reduce the semantic ambiguity of the recovered modalities, the available modalities are embedded as conditions in the recovering process so that we can recover semantically consistent samples from a distribution conditioned on the available modalities. Finally, to perform the MER, the multimodal fusion and prediction part receives recovered complete multimodal data and uses multimodal transformers to fuse multimodal representation for emotion recognition. The detail is introduced in the next subsections.
3.3 Shallow Feature Extractor
We consider three heterogeneous modalities: language  $(l)$ , vision  $(v)$ , and acoustic  $(a)$  for the MER task. Since the original dimensional spaces of the three modalities are often inconsistent, they are not suitable for direct use in missing modality recovering. To address this problem, we design a shallow feature extractor that contains three independent encoders  $\mathcal{E}_k$ ,  $k \in \{l, v, a\}$ , to extract the shallow features of three modalities and map them into the same dimensional space. Each encoder contains a 1D convolutional layer. Thus, the subsequent recovery task aims to recover missing modalities from the distribution space of shallow features.