Leveraging Knowledge of Modality Experts for Incomplete Multimodal Learning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 12:01:06

Abstract: Multimodal Emotion Recognition (MER) may encounter incomplete multimodal scenarios caused by sensor damage or privacy protection in practical applications. Existing incomplete multimodal learning methods focus on learning better joint representations across modalities. However, our investigation shows that they are lacking in learning the unimodal representations which are rather discriminative as well. Instead, we propose a novel framework named Mixture of Modality Knowledge Experts (MoMKE) with two-stage training. In unimodal expert training, each expert learns the unimodal knowledge from the corresponding modality. In experts mixing training, both unimodal and joint representations are learned by leveraging the knowledge of all modality experts. In addition, we design a special Soft Router that can enrich the modality representations by dynamically mixing the unimodal representations and the joint representations. Various incomplete multimodal experiments on three benchmark datasets showcase the robust performance of MoMKE, especially on severely incomplete conditions. Visualization analysis further reveals the considerable value of unimodal and joint representations.

Main paper: 
 introduction: 
 Multimodal learning [2, 31] , which leverages complementary information across different modalities, has achieved significant strides in automatic emotion recognition [11, 27, 37] . Recently, Multimodal Emotion Recognition (MER) has become one of the most popular research topics in affective computing with many applications, including human-computer interactions [5, 9] , dialogue systems [7, 13] and social media analysis [25] .Existing multimodal learning approaches in MER often implicitly assume that all modalities are available during training and testing. In the real world, however, factors such as sensor damage, automatic speech recognition errors, and privacy protection [36] often result in the unavailability of data from certain modalities. Recent studies [1, 18, 19] have shown that existing multimodal models trained on complete multimodal data are not inherently robust to incomplete multimodalities. Therefore, the development of a model capable of adapting to incomplete multimodalities constitutes a work of significant practical import at the present juncture.The most intuitive approach to address incomplete multimodalities is to impute missing modalities, among which variational autoencoders (VAEs) [24, 30] , generative adversarial networks (GANs) [4] , and diffusion models [29] have been demonstrated to be capable of imputing missing modalities based on available ones. However, the quality of the imputed modalities highly depends on the architecture and parameters of the generative networks [14] , and their high computational cost also limits their applications in scenarios requiring real-time processing [29] . Thus, an alternative approach, namely joint representation learning was widely accepted, which focuses on learning consistent joint representations across modalities. Essentially, all such methods follow a similar structural design, as shown in Fig. 1(a) , where encoders encode available modalities into a joint space, while decoders constrain the learning of the joint representations through modality reconstruction only during training. Among these, Pham et al. [19] learned joint representations using a Seq2Seq model and cyclic translation loss. Zhao et al. [36] designed an encoder-decoder structure based on cascaded residual autoencoders and constrained joint representation learning through cycle consistency learning. Zuo et al. [39] further extracted Although these methods have achieved notable success to date, their performance is markedly suboptimal in severely incomplete conditions, i.e., only unimodal data, especially only visual or audio modality is available, as shown in Tab. 1. This phenomenon can be attributed to the neglect of learning unimodal representations, which in turn leads to insufficient discriminability in unimodal conditions.Different from previous works, we explicitly define the unimodal and joint representations of the available modalities under incomplete multimodalities and propose the Mixture of Modality Knowledge Experts (MoMKE) with two-stage training to learn both of them, as illustrated in Fig. 1(b ). Specifically, MoMKE first learns unimodal knowledge separately through unimodal expert training, enabling each expert to obtain discriminative unimodal representations. Then for each modality, MoMKE further learns the joint representations by leveraging the knowledge of other modality experts, and dynamically mixes them according to varied inputs through a proposed Soft Router to achieve more delicate representation fusion. We call this stage experts mixing training. In this way, MoMKE obtains more comprehensive representations for available modalities under incomplete multimodalities. Experiments on three MER datasets showcase the considerable effectiveness of MoMKE, especially, for severely incomplete conditions. Ablation studies and visualization analysis of expert loads further validate the significant value of unimodal representations and the necessity of joint representations under incomplete multimodalities. The main contributions of this paper can be summarized as follows:  
related work 2.1 incomplete multimodal learning in mer: 
 Learning from incomplete multimodalities is an important research topic in machine learning, with significant implications for applying models to complex real-world scenarios. One direct approach to address incomplete multimodalities is data imputation. Unsupervised imputation methods include zero imputation and average imputation [16, 18, 34] . Recently, deep learning based methods have utilized missing modalities as supervision to generate them through generative models such as VAE [24, 30] and GAN [4] . Tran et al. [26] proposed the cascaded residual autoencoder (CRA), which stacks residual autoencoders to simulate the correlations between different modalities, thereby imputing missing modalities from available ones. Wang et al. [29] leveraged a score-based diffusion model to map random noise to the distribution space of missing modalities, and used available modalities as semantic conditions to guide the denoising process for recovering missing modalities. However, the high computational cost required by the imputation methods limits their application in real-time processing scenarios. Thereby methods that learn joint representations across modalities through consistency constraints are more widely used in incomplete multimodal learning for MER. Pham et al. [19] proposed the Multimodal Cyclic Translation Network (MCTN), learning joint representations through cyclic translation loss on Seq2Seq models. Zhao et al. [36] introduced the Missing Modality Imagination Network (MMIN) to learn multimodal joint representations through cross-modal imagination of missing modalities and cyclic consistency loss. Zuo et al. [39] further proposed IF-MMIN, which extracts invariant features between modalities based on central moment discrepancy distance. Luo et al. [15] proposed the Multimodal Reconstruction and Alignment Network (MRAN), enhancing model robustness by projecting visual and audio features into the text feature space. Lian et al. [12] proposed the Graph Completion Network (GCNet), utilizing the graph neural network for multimodal interaction to learn cross-modal representations. All these methods focused on learning joint representations across modalities through model design and constraints like reconstruction loss, but they overlooked the learning of the unimodal representations that contain modality-specific information.In contrast to the aforementioned approaches, our method introduces unimodal expert training to learn unimodal representations, which are then effectively mixed with joint representations through a novel expert mixing training. This significantly enhances the discriminative power of the representations used for inferences.In recent years, the Mixture of Experts (MoE) has been widely studied as a method to increase the model's capacity in parameter size without adding computational cost in natural language processing [6, 10, 22] and computer vision [17, 20, 23] . It typically consists of 𝑁 identical expert models and a router for assigning experts.The router 𝑟 is a trainable gating function that assigns a score to each expert 𝑒 based on the input 𝑋 . It then sparsely selects 𝐾 experts with top 𝐾 scores and uses a softmax function to compute the probability distribution of the outputs from the selected expert networks. Equation (1) mathematically represents MoE where 𝑂 is the output embeddings.EQUATIONIn this way, MoE achieves specialization among different experts and sparsity in the model.For the incomplete multimodal learning in MER, we focus on mixing the knowledge from all modality experts to enrich the modality representations instead of achieving sparsity of the model. Therefore, all modality experts are involved and routed, i.e., 𝐾 = 𝑁 , where each expert is pre-trained on the corresponding modality to learn unimodal knowledge. Hereinafter, this mechanism is referred to as the Soft Router, which is configured to dynamically mix the unimodal and joint representations under incomplete multimodalities. Its outputs (weights) indicate the importance of different representations for fusion. 
method 3.1 preliminaries: 
 In this subsection, we first formulate the problem of the General Incomplete Multimodal Learning in MER, then propose our new ideas to address the problem.Pre-define. Following previous work [12] , we consider incomplete multimodal learning in conversations, where audio 𝑎, text 𝑡 and visual 𝑣 modalities are utilized. Let's denote a conversation as 𝐺 = {(𝑢 𝑖 , 𝑦 𝑖 )} 𝐿 𝑖=1 , where 𝐿 is the number of utterances in the conversation, 𝑢 𝑖 is the 𝑖 𝑡ℎ utterance in 𝐺 and 𝑦 𝑖 is the label of 𝑢 𝑖 . When under complete multimodalities, 𝑋 = {𝑋 𝑚 |𝑚 ∈ {𝑎, 𝑡, 𝑣 }} is the multimodal feature set, where 𝑋 𝑚 = {𝑥 𝑚 𝑖 } 𝐿 𝑖=1 is the unimodal feature set. When under incomplete multimodalities, 𝑋 𝑀 = {𝑋 𝑚 |𝑚 ∈ 𝑀 } denotes the feature set of available modalities, and𝑋 M = {𝑋 m | m ∈ M }denotes the feature set of missing modalities, where 𝑀 ∪ M = {𝑎, 𝑡, 𝑣 } and 𝑀 ∩ M = ∅. 𝑌 = {𝑦 𝑖 } 𝐿 𝑖=1 denotes the label set. Please note that for general incomplete multimodal learning, all modalities are available during training. The missing modality 𝑋 M is not included in the inputs during testing, but can be used as supervision to train the model.General formulation. Here, we summarize the existing general paradigm of incomplete multimodal learning in MER. As shown in Fig. 1(a) , these methods typically consist of the following components: an encoder 𝐸 : 𝑋 𝑀 → 𝑅 that learns the mapping from available modalities to their representations, a classifier or regressor 𝐶 : 𝑅 → 𝑌 that maps the modality representations to labels, and a decoder 𝐷 : 𝑅 → 𝑋 M that maps the modality representations to missing modalities. The optimization objective is:EQUATIONwhere 𝑙 𝑡𝑎𝑠𝑘 is the task loss to constrain the learning of task-related representations and 𝑙 𝑟𝑒𝑐𝑜𝑛 is the reconstruction loss that utilizes 𝑋 M as supervision to constrain the learned representations to lie in the joint space. This enables the encoder 𝐸 to extract the joint representations that are shared between the available and missing modalities, i.e.,EQUATIONSome methods may also impose additional loss constraints while their purpose remains to learn consistent joint representations across modalities. However, our investigation shows that existing incomplete multimodal learning methods perform poorly in severely incomplete (unimodal) conditions. The possible reason is that the joint learning methods overlook the learning of the unimodal representations, that are not shared by other modalities, and the learned joint representations are less discriminative for the task. This raises a new question: how can one ensure the acquisition of a joint representation while simultaneously capturing unimodal representations under incomplete multimodalities?To address this issue, we redefine the unimodal and joint representations in the framework of our approach.Proposed formulation. In incomplete scenarios, there could be only one available modality, which is rather challenging to obtain joint representations through cross-modal interactions. Therefore, given a modality 𝑚, we distinguish its representations into unimodal and joint representations based on whether knowledge from other modalities is utilized during representation learning. Specially, given a unimodal feature set 𝑋 𝑚 , i.e. 𝑀 = {𝑚}, 𝑚 ∈ {𝑎, 𝑡, 𝑣 }, we define the unimodal representation as 𝑅 𝑚 𝑢𝑛𝑖 learned only from modality 𝑚, i.e.,EQUATIONwhere 𝑒 𝑀 (•) is an encoding function with the knowledge of modality 𝑚. We define the joint representation as 𝑅 𝑚 𝑗𝑜𝑖𝑛𝑡 learned through the knowledge from other modalities M, i.e.,EQUATIONwhere 𝑒 M (•) is an encoding function with the knowledge of other modalities M. Due to the complexity of various incomplete multimodalities, modality representations of a unimodal feature 𝑋 𝑚 become a mixed form. Thus for the 𝑋 𝑚 , its modality representations should be represented asEQUATIONwhere 𝛼 and 𝛽 are the weights of unimodal and joint representations determined by the inputs, and ⊕ denotes weighted fusion. Note that when the number of modalities exceeds two, 𝑒 M can be composed of multiple encoding functions with knowledge of different modalities. For example, in our scenario, when 𝑚 = 𝑎, 𝑅 𝑎 can be represented asEQUATIONIn this work, the objective of incomplete multimodal learning becomes to learn an encoder 𝐸 that encodes comprehensive modality representations 𝑅 𝑚 , i.e.,EQUATIONDifferent from learning the encoder in Eq. (3), Eq. ( 8) also requires learning the unimodal representation and its weights with the joint representations to enrich the modality representations. In addition, compared to learning unimodal representation only, the joint representations in Eq. ( 8) learned by leveraging knowledge from other modalities allow the model to achieve a more comprehensive understanding in unimodal conditions, which is validated in Section 4.4 and 4.6.To achieve Eq. ( 8), we proposed the Mixture of Modality Knowledge Experts (MoMKE) splitting the encoder into several modality knowledge experts and a Soft Router, i.e.,EQUATIONwhere 𝑒 𝑘 (•) is the expert with the knowledge of modality 𝑘, which is initialized by unimodal expert training on the corresponding modality. When 𝑘 = 𝑚, 𝑒 𝑘 (𝑋 𝑚 ) = 𝑅 𝑚 𝑢𝑛𝑖 , otherwise 𝑒 𝑘 (𝑋 𝑚 ) = 𝑅 𝑚 𝑗𝑜𝑖𝑛𝑡 . 𝑟 (•) is the fusion weights (𝛼, 𝛽 in Eq. ( 8) obtained by the Soft Router learning). The optimization objective is:EQUATIONEQUATIONThe overall framework of Mixture of Modality Knowledge Experts (MoMKE) is shown in Fig. 2 . For each modality, utterancelevel feature extraction is first performed. Then, the unimodal expert training (the blue block) trains each modality expert using its own modality data to learn unimodal knowledge. Afterward, in the most crucial experts mixing training (the orange blocks), all modality knowledge experts are trained to learn unimodal and joint representations, which will be further dynamically mixed by a Soft Router to obtain more comprehensive modality representations for various incomplete multimodal conditions.For the audio modality of each utterance, the pre-trained wav2veclarge [21] model is used to extract 512-dimensional audio features.For the text modality, the pre-trained DeBERTa-large [8] model is utilized to encode each utterance into 1,024-dimensional text features. For the visual modality, the MTCNN [35] face detection algorithm is exploited to extract aligned faces, followed by facial feature extraction using the pre-trained MA-Net [38] . The framelevel features are compressed into 1,024-dimensional utterancelevel features. Finally, for each utterance 𝑢 𝑖 , we extract multimodal features 𝑥 𝑖 = {𝑥 𝑚 𝑖 ∈ R 𝑑 𝑚 |𝑚 ∈ {𝑎, 𝑡, 𝑣 }}, where 𝑥 𝑎 𝑖 , 𝑥 𝑡 𝑖 and 𝑥 𝑣 𝑖 are the respective utterance features of audio, text and visual modality and 𝑑 𝑎 , 𝑑 𝑡 and 𝑑 𝑣 are the feature dimensions corresponding to each modality.For each expert, we employ a transformer encoder structure. Given unimodal features 𝑋 𝑚 = {𝑥 𝑚 𝑖 } 𝐿 𝑖=1 ∈ R 𝐿×𝑑 𝑚 , a projection matrix is applied to project them into a fixed dimension 𝑑:EQUATIONwhere 𝑊 𝑚 ∈ R 𝑑 𝑚 ×𝑑 , 𝑍 𝑚 ∈ R 𝐿×𝑑 . Then, the position embeddings 𝑍 𝑝𝑜𝑠 [28] are added to the features to encode the positional information of each utterance in the conversation:EQUATIONwhere 𝑍 𝑝𝑜𝑠 and 𝐻 𝑚 0 ∈ R 𝐿×𝑑 . The encoded features will be fed into the modality transformer expert composed of 𝑄 identical blocks, where each block consists of a multi-head self-attention (MSA) mechanism and feed-forward neural networks (FFN). Layer normalization (LN) and residual connections are applied within each block. The 𝑗-th transformer block can be described as follows:EQUATIONThe final output 𝐻 𝑚 𝑄 of the modality expert transformer will be processed through independent fully-connected layers to obtain labelsEQUATIONwhere 𝐻 𝑚 𝑄 ∈ R 𝐿×𝑑 , 𝜃 𝑚 𝑇 and 𝜃 𝑚 𝐹𝐶 represent the parameters of the transformer expert and fully-connected layers, respectively. The transformer expert is optimized by minimizing the task-specific loss, such as cross-entropy lossEQUATIONfor emotion recognition, or mean square error (MSE) loss, denoted asEQUATIONis used for sentiment analysis. After unimodal expert training, each modality expert has learned the unimodal knowledge, equivalent to the expert 𝑒 𝑚 (•) in Eq. ( 9), i.e., 𝑇 𝑟𝑎𝑛𝑠 𝑓 𝑜𝑟𝑚𝑒𝑟 𝜃 𝑚 𝑇 (•) = 𝑒 𝑚 (•)(𝑚 ∈ {𝑎, 𝑡, 𝑣 }).After unimodal expert training, each modality expert has learned the ability of extracting unimodal representations. At this stage, our objective is to learn both unimodal and joint representations based on all modality knowledge experts and dynamically mix them to better cope with various incomplete multimodal conditions. Taking the case where only audio modality is available as an example, i.e. 𝑚 = 𝑎, the audio features will be processed by both the audio expert and the experts with knowledge of other modalities to obtain the unimodal and joint representations of audio:EQUATIONwhere 𝑅 𝑎 𝑎 is the audio unimodal representations corresponding to 𝑅 𝑎 𝑢𝑛𝑖 in Eq. ( 8) (𝑚 = 𝑎), 𝑅 𝑎 𝑡 and 𝑅 𝑎 𝑣 are the joint representations of audio features incorporating text and visual knowledge corresponding to 𝑅 𝑎 𝑗𝑜𝑖𝑛𝑡 in Eq. ( 8) (𝑚 = 𝑎). It is analogous when text or visual modality is available.Due to the complexity of incomplete multimodalities, such as varied data qualities, tasks, and so on, the unimodal representations and joint representations may largely change. A fixed fusion strategy (such as a simple weighted average) can hardly achieve optimal results in all cases. Therefore, a Soft Router 𝑟 , i.e., a two-layer Multilayer Perceptron (MLP), is further introduced to dynamically measure the importance (weights) of the unimodal and joint representations according to different inputs, which will be used to mix the unimodal and joint representations to obtain the audio representation 𝑅 𝑎 :EQUATIONEQUATIONEQUATIONwhere 𝑆 𝑎 ∈ R 𝐿×3 , 𝑅 𝑎 ∈ R 𝐿×𝑑 , 𝑤 𝑎 𝑖 is the weights corresponding to 𝛼 and 𝛽 in Eq. ( 8) and 𝑅 𝑎 corresponds to the modality representations 𝑅 𝑚 . By observing how the Soft Router dynamically adjusts the weight of the representations, deeper insights into the roles of unimodal and joint representations can be gained. This is evaluated in the Section 4.5.The audio representation 𝑅 𝑎 , which mixes knowledge from all modality experts, is fed into a fully connected layer to obtain labels:EQUATIONThe optimization objective is given by Eq. ( 11), where the loss is either Eq. ( 16) or Eq. ( 17). When more modalities are available, the features of each modality still follow the above forward process to obtain its representation 𝑅 𝑚 . All modality representations are concatenated in the feature dimension to obtain a fused representation before inference. During the training process, the Soft Router and all the experts are trainable to learn the corresponding knowledge for extracting and mixing modality representations in various incomplete multimodal conditions. 
experiment 4.1 datasets and evaluation metrics: 
 To assess the effectiveness of MoMKE, we conduct experiments on three benchmark multimodal emotion recognition or sentiment analysis datasets, including IEMOCAP, CMU-MOSI, and CMU-MOSEI.IEMOCAP dataset [3] consists of 5 dyadic dialogue sessions, where actors perform improvisations or scripted scenarios. Each dialogue is further segmented into numerous utterances, with each utterance annotated with categorical emotion labels. For a fair comparison, we follow previous works [12, 36, 39] to form the fourclass emotion (happy, sad, neutral and angry) recognition setup.CMU-MOSI dataset [32] is a collection of 2199 opinionated video clips collected from YouTube, where 1284, 229, and 686 of them are used as training, validation, and test set. Each video clip is labeled with a sentiment score ranging from -3 (strongly negative) to +3 (strongly positive).CMU-MOSEI dataset [33] contains 22856 video clips from over 1000 online YouTube speakers, where 16326 of them are used for training, the remaining 1871 and 4659 of them are used for validation and testing. All utterances are randomly selected from a variety of topic and monologue videos and follow the annotation scheme of [-3, 3] in CMU-MOSI.For the IEMOCAP dataset, we adopt weighted accuracy (WA) and unweighted accuracy (UA) as evaluation metrics. For the CMU-MOSI and CMU-MOSEI datasets, we focus on negative/positive classification tasks in line with previous works [12, 15, 29] where negative and positive classes are assigned for < 0 and > 0 sentiment scores, and utilize ACC and F1 scores as evaluation metrics.Table 2 : Comparison with state-of-the-art (SOTA) methods on three benchmark datasets under all possible incomplete testing conditions (e.g. testing case {𝑎} means that only audio modality is available and both text and visual modalities are missing). "Average" refers to the average performance over all six incomplete multimodal conditions. The overall best results of each dataset are highlighted in red bold, while the second-best results are in black bold. The results with 𝑎 are from [29] , and the others can be found in their paper. The row with Δ 𝑆𝑂𝑇 𝐴 means the improvements or reductions of MoMKE compared to the best competing methods.The model structure remains the same as MoMKE, but the three transformers are randomly initialized instead of modality knowledge experts. 2) Without experts mixing training: Modality knowledge experts corresponding to missing modalities will not be used for representations mixing, i.e., only the modality experts corresponding to the available modalities are utilized. In this setting, for the unimodal conditions, the model is the corresponding modality transformer expert, while for the multimodal conditions, the inference is obtained by averaging the outputs of each availablemodality transformer expert. 3) Without router: The model structure is the same as MoMKE, but without the Soft Router, i.e., the weights of all experts are equal and the modality representation is the sum-average of all representations.The results of the ablation experiments are shown in Tab. 3. When unimodal expert training is removed, the performance drops significantly across all metrics. This highlights the importance of building unimodal knowledge experts, which can help to learn discriminative unimodal representations and further aid joint representation learning. A similar degradation can be observed when experts mixing training is removed, which demonstrates that the knowledge from other modality experts can help to learn more comprehensive representations, thereby improving the performance of MoMKE. Compared with the above two ablations, the performance of the model in Without router is better since it can learn both the unimodal and joint representations. However, there is still a significant gap compared to MoMKE, which suggests that the contributions of unimodal representations and joint representations inherently vary in diverse incomplete scenarios. Nonetheless, a well-trained Soft Router is able to dynamically adjust the weights to mix these representations according to the particularities of the incomplete scenario, thereby enhancing the modality representations.Ablation of the experts: To explore the impact of each modality expert on learning modality representations, we conduct experiments using different combinations of modality experts during experts mixing training. Table 4 shows the experimental results in the severely incomplete (unimodal) conditions on IEMOCAP. We divide the experiment settings into two groups: one in which the expert corresponding to the available modality is used in MoMKE (with underline), and the other in which the corresponding expert is not used (without underline). Results under each testing condition, i.e., in each column, show a clear conclusion that the models using the expert of the corresponding modality outperform those that do not use it. This validates our hypothesis that unimodal representations play a more crucial role in severely incomplete conditions. Furthermore, it can be observed that, based on utilizing unimodal representations, mixing more experts can achieve better performance. This highlights the importance of joint representations and shows that mixing the knowledge of more modality experts can help to obtain a more comprehensive modality representation.To have an insight into the role of unimodal and joint representations in severely incomplete conditions, we visualize the expert loads of the test set during experts mixing training, i.e., the weight of each modality expert assigned by the Soft Router. As shown in Fig. 3 time, the joint representations also account for a certain portion of the modality representations, which demonstrates their necessity.To further illustrate the necessity of learning joint representations through other modality knowledge experts, we carry out a case study by selecting a conversation from the IEMOCAP dataset, then visualize the expert loads in MoMKE when only text modality is available, and compare the inferences of using only text expert and MoMKE, as shown in Fig. 4 . It can be observed that when the model relies solely on text expert with the text unimodal representations, the inferences tend to be neutral or sadness, as indicated by the black dashed box. In contrast, MoMKE can leverage the knowledge from all modality experts to do correct inferences by mixing more joint representations from other modality experts, imagining the speakers' tone and expression from the text as a human would, thus obtaining a more comprehensive modality representation. 
conclusion: 
 In  
 
 Appendix:  


Decision: All the reviewers gave positive ratings and tend to accept the paper. SAC and AC agree with reviewers and recommend acceptance of the paper.

Reviews: review: paper_summary: paper_summary: The paper proposed a novel incomplete multimodal learning framework, named MoMKE, to solve the problem of lacking in learning the unimodal representations due to focusing on learning better joint representation. The proposed framework consists of two-stage training, which is unimodal expert training and experts mixing training with Soft Router to dynamically "mix" the unimodal representations and joint representations. The effects of the proposed framework are verified with three popular multimodal sentiment analysis or emotion recognition benchmarks, IEMOCAP, CMU-MOSI, and CMU-MOSEI, as well as ablation analysis for demonstration of each module’s effectiveness.
strength_weakness: strength_weakness: [S1] The related works are well-structured to understand the limitations of previous multimodal fusion works in incomplete conditions, as well as their contributions compared to previous works.


[S2] They carefully defined unimodal, joint representation and their proposed scenarios using dedicated formulations.


[S3] The ablation study is well-designed, especially for the case study in this paper, which effectively demonstrates why mixing the modality experts is necessary for multimodal emotion recognition.
limitations: limitations: [W1] Technical contribution is unclear. It is a low contribution if the proposed framework is just similar to the combination of the MoE-based fusion model[1] and SMEAR[2] that use a mixture of experts for the fusion of each modality and dynamic merging of experts with adaptive routing, respectively. The paper needs to describe the key differences from the naive solution that combines existing methods.


[W2] The baseline is weak to validate the performance of the proposed model. MTMSA[3] outperforms state-of-the-art methods, and the proposed framework at the IEMOCAP benchmark, and MTMSA reported performance trends according to the missing modality rate. 


[W3] Writing needs improvement. Section 3.1 is overly detailed, especially lines 298-306, which unnecessarily repeat information from the introduction.

[1] Goyal, Ankit, et al. "A multimodal mixture-of-experts model for dynamic emotion prediction in movies." 2016 ieee international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2016.
[2] Muqeeth, Mohammed, Haokun Liu, and Colin Raffel. "Soft merging of experts with adaptive routing." arXiv preprint arXiv:2306.03745 (2023).
[3] Liu, Zhizhong, et al. "Modality translation-based multimodal sentiment analysis under uncertain missing modalities." Information Fusion 101 (2024): 101973.
paper_summary: paper_summary: The paper presents a novel framework named Mixture of Modality Knowledge Experts (MoMKE) designed to address the issue of incomplete multimodal scenarios in Multimodal Emotion Recognition (MER). The proposed approach involves a two-stage training strategy: unimodal expert training and experts mixing training. This framework aims to enrich modality representations by leveraging both unimodal and joint representations through a Soft Router mechanism. Experimental results on three benchmark datasets (IMEOCAP, CMU-MOSI, CMU-MOSEI) demonstrate the robustness and effectiveness of MoMKE, particularly under severely incomplete conditions.
strength_weakness: strength_weakness: Novelty: The introduction of the Mixture of Modality Knowledge Experts (MoMKE) framework is a novel approach to handling incomplete multimodal learning. The two-stage training strategy effectively addresses the limitations of existing methods that focus solely on joint representations. 

Technical Correctness: The framework is well-justified with a solid theoretical foundation. The use of a Soft Router to dynamically mix unimodal and joint representations is technically sound and innovative. The ablation study in Table 3 presents the effectiveness of the designed modules, especially the Soft Roter.

Evaluation: The experiments are thorough and cover various incomplete multimodal scenarios. The results showcase significant improvements over state-of-the-art methods, particularly in severely incomplete conditions, as shown in Table 2.

Clarity: The paper is well-written and structured, making the concepts and methodologies clear and easy to follow. The inclusion of visualization analysis further aids in understanding the impact of the proposed approach.
limitations: limitations: Assumption on Training Data: Like previous works, the framework in this work still assumes that all modality data are available during training, which might not always be practical in real-world scenarios. Future work could explore scenarios with incomplete modality training data.

Unfair comparison: This paper proposes to use pre-trained wav2veclarge, pre-trained DeBERTa-large and pre-trained MA-Net for audio, text and visual features, respectively. These pre-trained models are not used in other works at the same time. The improvement in this paper could be mainly introduced by the more powerful unimodal encoders instead of the proposed framework. The author could try to produce some results with the common unimodal encoders, or no pretrained encoders.
paper_summary: paper_summary: Multimodal Emotion Recognition (MER) may encounter incomplete multimodal scenarios caused by sensor damage or privacy protection in practical applications. Existing incomplete multimodal learning methods focus on learning better joint representations across modalities. However, author's investigation shows they are lacking in learning the unimodal representations which are rather discriminative as well. This paper proposes a framework for MER task.
strength_weakness: strength_weakness: This paper proposes a novel framework Mixture of Modality Knowledge Experts (MoMKE) with two-stage training. In the unimodal expert training, each expert learns the knowledge of the unimodal from the corresponding modality. In experts mixing training, both unimodal and joint representations are learned by leveraging the knowledge of all modality experts. The experimental results increase greatly.
limitations: limitations: (1) The proposed two-stage training model will inevitably increase a large number of model parameters and computational cost, whether you can give the changes of the number of parameters and the impact on the running time.
(2) In the case of missing modalities (e.g., only audio is available), whether the audio modality extracted and jointly represented by the unimodal expert of other modalities will have a noise effect on the features of the original audio processed by the unimodal expert.
(3) How the unimodal joint representation of the model will be handled if only one modality is missing. It is suggested that a relevant explanation be added to this situation.
paper_summary: paper_summary: This research centers on the Multimodal Emotion Recognition (MER) task, particularly addressing Incomplete Multimodal Learning within MER. The study introduces the Mixture of Modality Knowledge Experts (MoMKE) to both learn and blend modalities, overcoming the constraint of prior methods that solely rely on joint representations. Additionally, it presents a two-stage training approach involving unimodal expert training and expert mixing training, employing a Soft Router to combine the unimodal and joint representations dynamically.
strength_weakness: strength_weakness: 1. The proposed idea is novel and good, choosing an important problem to focus on and solve.

2. The paper is written well.

3. The experimental setup and analysis are good.
limitations: limitations: 1. No mention of the number of seeds used to finetune the models. It is not clear if average and standard deviation are being reported for the tests. Generally, one must evaluate multiple seeds and then report the average and standard deviation.

2. Previous works in MSA running on CMU-MOSI and CMU-MOSEI reported MAE as a main metric since these tasks are considered regression tasks (labeled from -3 to 3). You should consider adding these metrics and compare them with some other methods.

3. I can not find \hat{X}t and \hat{X}v in Experts Mixing Training in Figure 2.