Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition (2024)

Abstract: The development of multimodal models has significantly advanced multimodal sentiment analysis and emotion recognition. However, in real-world applications, the presence of various missing modality cases often leads to a degradation in the model’s performance. In this work, we propose a novel multimodal Transformer framework using prompt learning to address the issue of missing modalities. Our method introduces three types of prompts: generative prompts, missing-signal prompts, and missingtype prompts. These prompts enable the generation of missing modality features and facilitate the learning of intraand inter-modality information. Through prompt learning, we achieve a substantial reduction in the number of trainable parameters. Our proposed method outperforms other methods significantly across all evaluation metrics. Extensive experiments and ablation studies are conducted to demonstrate the effectiveness and robustness of our method, showcasing its ability to effectively handle missing modalities. Codes are available at https://github.com/zrguo/MPLMM.

heading: 1 Introduction
text: Humans perceive the world in a multimodal way, such as sight, sound, touch and language. These multimodal features can provide comprehensive information to help us understand and explore the world. Thus, modeling and mining multimodal data is of great importance and has much potential. Recently, multimodal sentiment analysis (Tsai et al., 2019; Hazarika et al., 2020; Han et al., 2021; Hu et al., 2022) has attracted much attention. However, there are two main challenges in many existing methods: 1) Different from common multimodal tasks which only have two modalities (image and text), multimodal sentiment analysis task often has more modalities (video, audio, text, etc.). There-
∗ Corresponding author
fore, in real-world scenarios, missing modality conditions always occur due to equipment failure, data corruption, privacy issues and the like, especially in low-resource domains, which could lead to a degradation in the model’s performance. Current multimodal models trained on complete data usually fail when tested on incomplete data (Aguilar et al., 2019; Pham et al., 2019). 2) With the success of large-scale multimodal models (Kim et al., 2021; Li et al., 2021; Radford et al., 2021), lots of researchers tend to finetune these large pre-trained models to downstream tasks. However, this kind of finetuning is infeasible for many researchers because it requires large computational resources. Besides, finetuning such a pre-trained model on small datasets could lead to instability (Mosbach et al., 2021).
Recently, prompt learning (Gao et al., 2021; Heinzerling and Inui, 2021; Khattak et al., 2023; Lee et al., 2023) is proposed, which freezes all the parameters of a pre-trained model while only finetuning several prompts and it has achieved great success (Lester et al., 2021). Motivated by prompt learning, in this paper, we intend to exploit a highresource dataset that contains relatively more complete modality data for pre-training and then leverage several trainable prompts to transfer the knowledge from high-resource domains to low-resource domains where missing modality cases often occur.
Previous works (Ma et al., 2021; Pham et al., 2019; Zhao et al., 2021) mainly focus on introducing sophisticated architecture to address the issue of missing modalities. These methods do not use pretrained models and usually require a lot of computational resources. However, our method is based on prompt learning, which only finetunes a few parameters of prompts. Lee et al. (2023) is a recent work which is similar to ours. However, its proposed missing-aware prompts increase exponentially with the number of modalities. In contrast, our proposed prompts increase linearly with the number
1726
of modalities which is more parameter-efficient. Specifically, we propose three types of prompts: generative prompts, missing-signal prompts, and missing-type prompts which can learn the representations of the missing modalities, cross-modal and fine-grained features. These three types of prompts play a combined role in improving the model’s performance.
We conduct extensive experiments on four datasets: CMU-MOSEI (Bagher Zadeh et al., 2018), CMU-MOSI (Zadeh et al., 2016), IEMOCAP (Busso et al., 2008) and CH-SIMS (Yu et al., 2020). The proposed method outperforms the baselines significantly across all metrics on all datasets. We further study the roles of three types of prompts, the effect of missing rate of training data, and the effect of prompt length. We find that: 1) missingsignal prompts are modality-specific while missingtype prompts are modality-shared which represent intra-modality and inter-modality information respectively. 2) with short prompts, our model can achieve very good results which demonstrates our proposed method is parameter-efficient. 3) the missing rate is important for the performance of the model, with 70% being the optimal value.
Our contributions can be summarized as follows: • We present a novel framework via prompt
learning for sentiment analysis and emotion recognition which is not only computationally efficient but also capable of handling missing modalities during both the training and testing stages. • The number of parameters of our proposed prompts is linearly related to the number of modalities, which significantly reduces computational resources. • We propose three types of prompts to address the issue of missing modalities. These three types of prompts can generate missing information, and learn intra- and inter-modality information respectively. • Our proposed method outperforms all the baselines across all metrics significantly. Furthermore, we discover that applying modality dropout with a rate of 70% during training yields the best enhancement in the model’s performance.

heading: 2 Related Works
text: Multimodal Sentiment Analysis (MSA) and Emotion Recognition (MER). Multimodal sentiment analysis and emotion recognition refer to the process of analyzing and understanding human sentiment or emotions using multiple modalities of data, such as text, image, audio, and video. The main challenge of such tasks is how to effectively use the information from different modalities to complement each other. Currently, there are two main multimodal fusion strategies: feature-level fusion and decision-level fusion. Feature-level fusion methods (Liang et al., 2018; Wang et al., 2019) combine features from different modalities to create a unified feature representation via concatenation or other methods. For example, Liang et al. (2018) decomposed the fusion problem into multiple stages and fused features step by step to obtain a comprehensive representation. Mai et al. (2019) conducted fusion hierarchically so that both local and global interactions are considered for a comprehensive interpretation of multimodal embeddings. Different from feature-level fusion methods, decision-level fusion methods (Tsai et al., 2019; Hazarika et al., 2020; Han et al., 2021; Hu et al., 2022) process different modalities independently and then incorporate them into the final decision. For instance, Tsai et al. (2019) proposed a directional pairwise cross-modal attention to implement modal alignment and fused the outputs of each modality at the decision level to make predictions. These methods all assume that the data is complete while our proposed method can deal with the situation when there exist missing modalities. Multimodal Learning with Missing Modalities. The presence of a missing modality poses challenges for multimodal learning because the model needs to effectively handle the absence of information while still making accurate predictions. Ma et al. (2021) proposed the SMIL model which leverages Bayesian meta-learning to address the issue of missing modalities. Some methods (Cai et al., 2018; Du et al., 2018) directly generate missing modalities using the available modalities. Zhao et al. (2021) proposed learning robust joint multimodal representations which can predict the representation of any missing modality given the available modalities. However, these methods always introduced sophisticated architecture to address the issue of missing modalities, which is computationally expensive. In comparison, our approach utilizes three different prompts to handle missing modalities, which is computationally more efficient. In a more recent work (Lee et al., 2023), prompts are used to address missing modalities, but the
number of prompts increases exponentially with the number of modalities. In contrast, the number of prompts in our method is linearly related to the number of modalities.
Prompt Learning. Prompt learning, which refers to the process of designing or generating effective prompts to use a pre-trained model for different types of downstream tasks, has been widely used in various NLP tasks (Gao et al., 2021; Heinzerling and Inui, 2021). With the success of prompt learning in NLP tasks (Lester et al., 2021; Li and Liang, 2021; Liu et al., 2022), recent works (Tsimpoukelli et al., 2021; Liang et al., 2022; Khattak et al., 2023) explored to leverage prompts in multimodal learning. Tsimpoukelli et al. (2021) presented a method for transforming large language models into multimodal systems by extending the soft-prompting philosophy of prefix tuning to ordered sets of images and texts. Khattak et al. (2023) proposed a strategy to ensure synergy between vision-language modalities by explicitly conditioning the vision prompts on textual prompts across different Transformer stages. More recently, Lee et al. (2023) proposed missing-aware prompts to address missing modalities which increase the robustness of the model, but it did not recover the missing information from the multimodal input. In comparison, our approach utilizes generative prompts to generate the representation of missing modalities given available modalities which can help further boost the performance of the model.

heading: 3 Proposed Method
text: In this section, we describe our proposed method (Figure 1) via prompt learning to address the issue of missing modalities (introduced in Section 3.1). Specifically, we introduce three kinds of prompts: generative prompts (introduced in Section 3.2), missing-signal prompts, and missing-type prompts (introduced in Section 3.3).

heading: 3.1 Overall Architecture
text: Problem Definition. Given a multimodal dataset D consisting of M = 3 modalities (e.g., audio, video and text), we use x = (xa, xv, xt) to represent a pair of features in D, where xa, xv, xt represent the features of acoustic, visual and textual modalities respectively. To indicate missing modalities, we use xam, xvm, xtm to denote which modalities are absent.
Figure 1 shows the overall architecture of our proposed model. For simplicity and better comparison, we use MulT (Tsai et al., 2019) as the backbone, which introduced the Crossmodal Transformer for modeling unaligned data. In our proposed method, we employ three types of different prompts: generative prompts, missing-signal prompts, and missing-type prompts. The generative prompts assist the available modalities in generating representations for the missing modalities. The missing-signal prompts are designed to inform the model about the absence of a specific modality while the missing-type prompts inform the model
about the absence of other modalities.

heading: 3.2 Missing Modality Generation Module (MMGM)
text: Many methods address missing modality issues by recovering missing information using available modalities (Cai et al., 2018; Du et al., 2018). However, these methods often utilize complex structures. Based on this observation, we propose the Missing Modality Generation Module (MMGM) which utilizes generative prompts to recover missing information in a much simpler way. We denote generative prompts as PG = (PGa, PGv, PGt) where PGa, PGv and PGt represent the generative prompts for the audio, video and text modalities, respectively. PG ∈ R3×dp×ℓp where dp and ℓp represent the dimension and length of the prompts respectively. Figure 2 illustrates the MMGM. Given x = (xam, xv, xt), we can generate the representation of the missing modality xam using the available xv and xt according to the following equation:
x̂a = fvt→â([PGa, fv→a(x v), ft→a(xt)]) (1)
where x̂a denotes the representation generated, [. . . ] represents the concatenation operation, f(·) represents a Conv block which consists of a Conv 1D layer and an activation function and → represents from one or two modalities to another modality. If there are two missing modalities, such as x = (xam, xvm, xt), the generation process is as follows:
x̂a = ft→â([PGa, ft→a(x t)]) x̂v = ft→v̂([PGv, ft→v(x t)])
(2)
After applying the MMGM, we can represent the generated features as x = (x̂a, x̂v, xt).

heading: 3.3 Missing-signal and Missing-type Prompts
text: MMGM recovers missing information using available modalities. However, the information generated sometimes might not be accurate and could mislead the model. Therefore, missing-signal prompts are designed to inform the corresponding Transformer whether the information for a particular modality is real or generated. For each modality, there are two missing-signal prompts: PMS to denote a modality is missing and PNMS to denote a modality is not missing. As depicted in Figure 1, after the MMGM and the Conv 1D layer, we obtain features x = (x̂a, xv, xt) where the audio modality is missing originally. We can incorporate the missing-signal prompts as follows:
x̂a := x̂a + P aMS xv := xv + P vNMS xt := xt + P tNMS
(3)
After applying missing-signal prompts, the model knows which modalities are generated and which modalities are real, which can help the model make better use of the recovered information. Notably, missing-signal prompts are modality-specific which means that this kind of prompt only considers a specific modality and does not take into account the correlations between the absence of multiple modalities. To address this limitation, we propose missing-type prompts.
If there are M modalities, there can be a total of 2M − 1 different cases of missing modalities. One intuitive approach is to design 2M − 1 prompts to handle each situation individually (Lee et al., 2023). However, as the number of modalities increases, this approach becomes computationally expensive. Therefore, we introduce a missing-type projection matrix MP. We can obtain MP of x = (xam, xv, xtm) as follows:
MP = Ma ·P aMS+Mv ·P vNMS+Mt ·P tMS (4)
where · is the matrix multiplication, Ma, Mv, Mt∈ Rdp×ℓp and Mp ∈ Rdp×dp . Then, we can get the missing-type prompts P ′MT as follows:
P ′MT = PMT ·MP (5)
where PMT represents the original missing-type prompts, P ′MT represents the projected missingtype prompts and PMT , P ′MT ∈ R3×ℓp×dp . In Figure 3, we illustrate how to attach the missingtype prompts to the Transformer with an example of x = (xam, xv, xtm). For each data pair in D, the corresponding missing-type modality prompt is attached according to the situation of missing modalities.

heading: 4 Experiments

heading: 4.1 Datasets and Evaluation Metrics
text: To simulate real-world scenarios, we select CMUMOSEI (Bagher Zadeh et al., 2018) as the highresource dataset while CMU-MOSI (Zadeh et al., 2016), IEMOCAP (Busso et al., 2008) and CHSIMS (Yu et al., 2020) are selected as the lowresource datasets. We pre-train our backbone on CMU-MOSEI and evaluate our proposed method on the four datasets. CMU-MOSI is a popular dataset for multimodal (audio, text and video) sentiment analysis, comprising 93 English YouTube. Each segment is manually annotated with a sentiment score ranging from strongly negative to strongly positive (-3 to +3). CMU-MOSEI is an extension of CMU-MOSI. It contains more than 65 hours of annotated video from more than 1000 speakers and 250 topics. Compared with CMU-MOSI, it covers a wider range of topics. IEMOCAP contains recorded videos from ten actors in five dyadic conversation sessions. It contains approximately 12 hours of data. Following previous works (Wang et al., 2019; Tsai et al., 2019),
four emotions (happiness, anger, sadness and neutral state) are selected for emotion recognition. CH-SIMS is a Chinese multimodal sentiment analysis dataset. It contains 2,281 video segments annotated with a sentiment score ranging from strongly negative to strongly positive (-1 to 1).
For CMU-MOSI and CMU-MOSEI, we follow previous works and adopt 7-class accuracy (ACC7), binary accuracy (ACC), F1 score (F1), mean absolute error (MAE) and Pearson correlation (Corr) as evaluation metrics. For IEMOCAP, we implement four binary classification tasks and use the average accuracy (ACC) and F1-weighted score (F1) as evaluation metrics. For CH-SIMS, we use binary accuracy (ACC), F1 score (F1), mean absolute error (MAE) and Pearson correlation (Corr).

heading: 4.2 Baselines
text: We compare our proposed method with the following methods: Lower Bound (LB) is trained with different combinations of modalities. Specifically, we train six different models using different combinations of modalities. Modality Substitution (MS) substitutes missing modality with a default value or a placeholder. Modality Dropout (MD) is a model trained with randomly dropped modalities during the training phase. MCTN (Pham et al., 2019) learns robust joint representations by translating between modalities to deal with missing information. MMIN (Zhao et al., 2021) learns robust joint multimodal representations, which can predict the representation of any missing modality given available modalities. MPMM (Lee et al., 2023) uses missing-aware prompts to instruct the model to address missing modality issues.

heading: 4.3 Implementation Details
text: Raw Feature Extraction. To demonstrate the generalization ability of our method, we implement three kinds of methods to extract features. For CMU-MOSEI and CMU-MOSI, we follow Tsai et al. (2019) to extract features. For IEMOCAP, we follow Zhao et al. (2021) to extract acoustic, visual and textual features. For CH-SIMS, we follow Yu et al. (2020) to extract features. Model Training Details. We first pre-train our backbone MulT (Tsai et al., 2019) on the CMUMOSEI dataset. Then, we freeze all the parameters of the backbone and only train several learnable prompts, Conv layers and the output layer (As shown in Figure 1). The length of prompts ℓp is set to 16 by default. We use L1 loss function for CMU-
MOSEI, CMU-MOSI and CH-SIMS datasets and cross-entropy loss for IEMOCAP dataset. In all experiments, we use Adam optimizer with a batch size of 64. We train the model for 30 epochs with a learning rate of 1× 10−3. Besides, we randomly discard the modality of the data with a missing rate of η = 70% during training. We fix the random seed to ensure that each model is trained on the same data.

heading: 4.4 Main Results
text: In Table 1, we present quantitative results on four datasets. The baselines LB, MS, MD and MPMM share the backbone of our method, thus reflecting the effectiveness of our proposed method to deal with missing modalities. Comparing the baseline MS and MD, we find that random discarding of data modalities during training improves the generalization ability of the model, thus making the model less sensitive to the data with missing modalities during the test phase.
Analyzing the results presented in the table, we observe that our proposed method outperforms the
baselines by a large margin in all datasets under all six missing modality cases. Additionally, our method brings great enhancement when text modality is missing, with 8-13% increase in accuracy compared with the LB baseline. This indicates that the three types of proposed prompts effectively guide the pre-trained model and yield impressive performance improvements.
Besides, it is worth noting that we implement different feature extraction approaches on four datasets. From the results in Table 1, we can see that our model outperforms the baselines on all datasets, which shows that our model can adapt to features extracted by different methods. This indicates that our model learns relative rather than absolute relationships between features, demonstrating its robustness and versatility.
In Figure 4, we further compare the performance of our model with other methods under different modality missing rates during test time. From the figure, we find that our model performs better than all the other methods across all metrics under different modality missing rates, although the model
is trained using the dataset with a modality missing rate of η = 70%. This indicates that our proposed method is robust to the missing rate of test set and can deal with severely missing modalities well.
Furthermore, the number of trainable parameters of our method is about 5-10% percent of that of the backbone. The majority of trainable parameters come from the Conv layers in MMGM. The number of trainable parameters of three types of prompts only accounts for 0.5-1% of that of the backbone. Notably, the number of trainable parameters does not increase with the size of the backbone network, which means that even if we use a much larger backbone, the number of trainable parameters remains the same. In all our experiments, we use only one 10 GB GPU (RTX 3080) with a batch size of 64. This demonstrates that our method is parameter-efficient.

heading: 4.5 Generalization Ability
text: To further validate the generalization ability of our method, we conduct experiments using different MSA/MER backbones. Specifically, we conduct experiments using MISA (Hazarika et al., 2020), MMIM (Han et al., 2021) and UniMSE (Hu et al., 2022) and present the results in Table 2. For all three backbones, we insert our generative prompts and module after the feature extractors. For UniMSE, we insert missing-signal and missingtype prompts into its multimodal fusion layers. For MMIM and MISA, we insert missing-signal and missing-type prompts into their modality-specific
encoders and fusion layers, respectively. The results in the table demonstrate our method can enhance the ability of various backbones to address missing modality issues. Besides, our prompts can enhance the performance in the complete data situation, indicating that our missing-signal and missing-type prompts can help the model learn intra-modality and inter-modality features.

heading: 4.6 Ablation Study
text: We divide our ablation experiments into three parts: contributions of three types of prompts, the effect of modality missing rate during training and the effect of prompt length. Contributions of three types of prompts. In Table 3, we present quantitative results of the contributions of different prompts. For CMU-MOSI, we can observe that generative prompts give the greatest improvement in ACC and F1, while missingsignal prompts improve the Corr the most and missing-type prompts improve the ACC-7 the most.
This indicates that generative prompts can help the available modalities generate the missing information which improves the binary accuracy. Besides, missing-type prompts tell the model whether other modalities are missing, thus strengthening the interactions between different modalities and learning cross-modal and fine-grained information which helps improve the ACC-7 a lot.
From the performance of models with different combinations of three types of prompts, we can further demonstrate the different roles of three types of prompts. We can conclude that generative prompts learn good representations of the missing modalities and improve the binary accuracy, missing-signal prompts are modality-specific prompts that tell models whether the corresponding modality is missing and help improve the correlation of the model’s predictions with humans, and missing-type prompts are shared prompts with inter-modality information, thus helping models learn cross-modal and fine-grained information that improves ACC-7. Furthermore, the combinations of three types of prompts further enhance the performance of the model on all datasets. This fully confirms the validity of our proposed method. Besides, we use an example in CH-SIMS to study the effectiveness of three types of prompts and present the results in Figure 5. From the figure, we can observe that the visual modality is key to predicting the correct result. With the prompts attached, the model can predict the accurate result, indicating the effectiveness of our method. The effect of modality missing rate during training. We study the impact of modality missing rate during training on the performance of the model in Figure 6. From the figure, we find that starting at a low point, both ACC and F1 score steadily improve as the train set modality missing rate increases, before reaching the highest point when the missing rate η = 70%. Then both ACC and F1 score decrease as the missing rate increases. This indicates that when the train set missing rate is low, it is difficult for a model to learn very good representations in the MMGM and to learn opportune prompts that can instruct the model well. This is because when the missing rate is low, the model tends to find a shortcut which to some degree prevents the model from learning good representations.
With the missing rate higher, the model has to learn how to generate missing information to make predictions more accurate. However, if the missing rate is higher than 70%, due to the amount of missing information, it is also hard for a model to learn good representations and prompts.
The effect of prompt length. To study the impact of prompt length on our model, we train our model on CMU-MOSI with nine different prompt lengths and present results in Figure 7. In the figure, we show the improved accuracy (IACC) over the baseline "Modality Dropout" of models with different prompt lengths. Intuitively, the longer the prompt length, the better the performance of the model. However, with the results shown in the figure, we find that when the prompt length ℓp = 16, the model performs the best. When the prompts are longer than 20, with the increase of the prompt length, the performance of the model decreases. Therefore, we deduce that it may be because our task is not complex and therefore the increase in parameters may overfit the model. Besides, we introduce parameter utilization rate ξ = IACC/ℓp to represent a trade-off between the performance of
models and the number of parameters of prompts. From the figure, we can clearly see that ℓp = 16 is the best choice, where IACC and ξ are both high compared with others. This also indicates that our proposed method can help improve the baseline with only a few parameters.

heading: 5 Conclusion
text: In this paper, we propose a novel multimodal Transformer via prompt learning to tackle the issue of missing modalities. We propose three types of prompts: generative prompts, missing-signal prompts, and missing-type prompts. Generative prompts can help generate missing information. Missing-signal prompts are modality-specific and missing-type prompts are modality-shared, which help the model learn intra-modality and intermodality relationships respectively. With prompt learning, we can significantly reduce the number of trainable parameters. Extensive experiments and ablation studies demonstrate the effectiveness and robustness of our proposed method.
Limitations
Our missing modality generation module generates missing information through two simple Conv blocks and generative prompts. This module improves the performance of our model significantly. However, we use extracted features but not raw features. Due to the simplicity of our missing generation module, the performance of the model could degrade if we use raw features which are much more complicated and have weaker correlation between modalities than extracted features. We leave this problem to future work.

heading: Acknowledgements
text: This work was supported by National Key R&D Program of China under Grant No.2022ZD0162000.

heading: Future Work
text: We leave this problem to future work