Enhanced Experts with Uncertainty-Aware Routing for Multimodal Sentiment Analysis

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-03 13:13:13

Abstract: Multimodal sentiment analysis, which has garnered widespread attention in recent years, aims to predict human emotional states using multimodal data. Previous studies have primarily focused on enhancing multimodal fusion and integrating information across different modalities while overlooking the impact of noisy data on the internal features of each single modality. In this paper, we propose the Enhanced experts with Uncertainty-Aware Routing (EUAR) method to address the influence of noisy data on multimodal sentiment analysis by capturing uncertainty and dynamically altering the network. Specifically, we introduce the Mixture of Experts approach into multimodal sentiment analysis for the first time, leveraging its properties under conditional computation to dynamically alter the network in response to different types of noisy data. Particularly, we refine the experts within the MoE framework to capture uncertainty in the data and extract clearer features. Additionally, a novel routing mechanism is introduced. Through our proposed U-loss, which utilizes the quantified uncertainty by experts, the network learns to route different samples to experts with lower uncertainty for processing, thus obtaining clearer, noise-free features. Experimental results demonstrate that our method achieves state-of-the-art performance on three widely used multimodal sentiment analysis datasets. Moreover, experiments on noisy datasets show that our approach outperforms existing methods in handling noisy data. Our anonymous implementation code can be available at https://anonymous.4open.science/r/EUAR-7BF6.

Main paper: 
 introduction: 
 In today's digital era, the collective influence of multiple modalities has garnered widespread attention, with multimodal learning emerging as one of the current research hotspots. Among these, Multimodal Sentiment Analysis (MSA), as a core task in multimodal learning, has also received significant interest from researchers. The former may occur due to the malfunction of perceptive devices and result in corrupted classification results. Meanwhile, the latter exists in the samples such as blurred images, ambiguous text, and so on, which produce low quality data and disrupt model's prediction tremendously.Most existing methods [1] [2] [3] [4] approach this task from the perspective of multimodal fusion, addressing the interaction of information across modalities at different levels such as early [5] , intermediate [6, 7] , and late fusion [8] , on raw data, feature, and decision levels respectively. Their efforts primarily focus on resolving information interaction among modalities, and these works have extensively demonstrated the effectiveness of fusion in the MSA task.However, prevalent approaches to multimodal sentiment analysis exhibit a notable limitation: they tend to overly prioritize intermodal information interaction, often neglecting the presence of noise within individual modalities and the diverse degrees of noise across different samples. We comprehend these two types of noise as modal-wise and sample-wise. Put simply, when confronted with noisy data, existing networks inadequately address this challenge. The processing of multimodal data inevitably entails encountering noise scenarios such as modality missing, image blurriness, and text ambiguity. As depicted in Figure 1 , scenarios such as audio interruptions, image distortions, and text ambiguities lead to a scenario where samples with ground truth as positive are misclassified by most networks as neutral or even negative. These noise instances can result in catastrophic consequences for network performance.Recently, some efforts [7, [9] [10] [11] have begun to address this issue by contributing solutions to modalities' noise scenarios like missing and blurriness. For instance, a recent work [9] quantifies the uncertainty level of the model regarding different modality data noise and incorporates uncertainty-weighted fusion. In [7] , Wang et al. work reconstructs data samples using multimodal flows to address modality missing scenarios. Although these methods have somewhat mitigated the impact of noise on networks, their performance still fall short when facing samples suffering from varying degrees of noise.Therefore, based on these challenges, we pose two questions: (1) How to quantify noise in samples to obtain clearer features? Inspired by [12] , we map different sample instances to a high-dimensional Gaussian distribution space and utilize sample variance to quantify inherent uncertainty. (2) How to dynamically adapt our network according to different noise scenarios? We introduce a Mixture of Experts (MoE) and utilize conditional computation to dynamically alter our network based on the noise scenario of different samples. Based on these considerations, we propose a novel MSA model in this paper, termed Enhanced experts with Uncertainty-Aware Routing (EUAR), addressing the shortcomings of previous works.Specifically, our work can be comprised of the following components: (1) Mapping samples to multimensional Gaussian distributions using different experts in EUAR, to quantify uncertainty using variance, and generate noise-free features using means. (2) Introducing the MoE structure to dynamically select different experts based on different samples, dynamically altering the network structure according to the scenario of different samples. (3) Training on basis of uncertainty-based dynamic routing algorithms, which enable the model to select experts with more confidence for specific samples to handle data.Overall, our contributions can be summarized as follows:• To our best knowledge, we are the first to introduce the MoE structure into multimodal sentiment analysis, dynamically altering the network to address noise within modalities and varying degrees of noise across different samples, thus complementing the shortcomings of previous works. • We devise a novel routing algorithm to enhance the experts, enabling the network to quantify uncertainty regarding noise and utilize this uncertainty to guide the routing algorithm in selecting experts with more confidence to process samples. • We conduct extensive experiments on three widely used multimodal sentiment analysis datasets, achieving state-of-theart performance and demonstrating superior performance under noise conditions compared to existing methods. 
related works 2.1 multimodal sentiment analysis: 
 Previous research aimed to find superior multimodal fusion methods to comprehensively integrate multimodal information for MSA task. These methods mainly focus on early fusion, feature fusion, late fusion, etc. For instance, graph fusion [13] based on feature fusion applies graph learning to integrate features, promoting fusion through constructing different nodes and integrating multimodal information. [9] employed a late fusion approach, merging decision levels by acquiring different weight proportions. These methods have achieved notable success in multimodal sentiment analysis. However, there are still some unsolved problems: both in video datasets and text-image datasets, the varying quality of video frames, images, and text poses challenges for these traditional methods. When encountering situations with low video quality, blurry images, and ambiguous text, the performance of these methods significantly diminishes, making it challenging to accurately identify the sentiment represented in the data. Recently, some work [14, 15] theoretically confirmed that noise in the data can bottleneck the model's performance. To address this issue, we propose the EUAR method, dynamically altering the network to effectively alleviate the bottleneck caused by data noise,i.e., modal-wise and sample-wise noise in sentiment analysis.In general, uncertainty in deep learning can be categorized into epistemic uncertainty and aleatoric uncertainty. The former represents the uncertainty inherent in the neural network itself, indicating the confidence level in its predictions. The latter refers to the inherent noise present in the data, such as image blurriness, text ambiguity, or missing modalities, which pose significant challenges to the model's predictions. In recent years,some efforts have been made to address uncertainty in conventional tasks like face recognition [12] ,image classification [16] and semantic segmentation [17] . It is worth noting that such uncertainty is also prevalent in multimodal sentiment analysis which impose negative impacts on final decisions. Due to the need for integrating multimodal data, if one modality is corrupted by noise, the overall multimodal features are correspondingly viciously affected. Recently, some research [7, 9, 10] has focused on addressing uncertainty in multimodal learning. However, these efforts primarily concentrate on integrating deterministic information across different modalities without considering the uncertainty within individual modalities. Therefore, we develop this work to fill this gap, aiming to overcome the information bottleneck caused by data noise in the MSA task.Mixture of Experts, a technique for expanding model parameters, has attached widespread attention in recent years alongside the rise of large models. It has been extensively explored in the fields of computer vision [18] [19] [20] and natural language processing [21] [22] [23] . MoE employs a learned gating mechanism to selectively activate different experts, adapting the network dynamically by activating k experts tailored to handle specific inputs. This dynamically alters the network without increasing additional computational overhead, thereby expanding the model's capacity. However, current research [24] [25] [26] has primarily focused on utilizing MoE to augment model parameters, neglecting the superiority of dynamic networks themselves in the context of conditional computation. Conditional computation [19, 22] selectively activates relevant parts of the model based on input-dependent factors, dividing the model into smaller, specialized sub-models. Different experts exhibit targeted effects for specific inputs. Therefore, we introduce MoE for the first time into the task of multimodal sentiment analysis, enhancing MoE's experts and designing an uncertainty-guided gating mechanism. This enables experts to capture uncertainty and dynamically selects corresponding experts based on the varying levels of noise in different inputs, addressing challenges posed by noise in multimodal sentiment analysis. 
proposed methods: 
 Our proposed EUAR method is a multimodal sentiment analysis framework that dynamically adapts its network structure based on i am so upset, why evan. different inputs. Our task is to predict the corresponding sentiment intensity values using given video clips or image-text pairs. Taking image-text pairs as an example, in one input sample, there are two different modal data:textual, and visual. As shown in Figure 2 , we first perform feature extraction in a specific manner and then use the extracted features in our proposed EUAR to predict the corresponding sentiment intensity labels.To align with prior research [1, 9] for fair comparison, we employed FACET, COVAREP, and BERT as feature extractors for the visual, audio, and text modalities, respectively, on the tri-modal dataset. For the bi-modal dataset, ResNet-152 and BERT are utilized as feature extractors for the visual and text modalities, respectively. The following narrative in this paper will be based on the tri-modal dataset. Given video clips, we feed the visual(𝑣), audio(𝑎), and text(𝑙) modalities into their respective feature extractors to obtain samples 𝑋 𝐵×𝐷 𝑚 , 𝑚 ∈ {𝑣, 𝑎, 𝑙 } required for this task, where 𝐵 represents batch size and 𝐷 represents feature dimensions. The objective is to integrate the features 𝑣, 𝑎, and 𝑙, and predict continuous sentiment intensity values 𝑦 ∈ 𝑅.In MoE technique, experts refer to a group of sub-models or neural networks, each responsible for processing different aspects or subspaces of the input data. These experts can be seen as "specialists" focusing on different tasks or data distributions. In previous works, experts were often defined as multi-layer perceptrons. Inspired by [12] , we enhanced the experts to capture uncertainty when dealing with samples influenced by noise to varying degrees. After feature extraction, we obtain multimodal features 𝑋 𝑚 𝑖 , 𝑚 ∈ {𝑣, 𝑎, 𝑙 },where i represents the instance. We map the features of each modality to a diagonal multivariate normal. Specifically, we define the representation 𝑧 𝑚 𝑖 of each sample 𝑥 𝑚 𝑖 in the latent space as a Gaussian distribution,which can be represented as:EQUATIONEQUATIONwhere 𝑚 represents different modalities, and 𝜃 represents the parameters in the neural network. For the parameters 𝜇 and 𝜎 of the Gaussian distribution, we employ a neural network to predict, where 𝑓 1 and 𝑓 2 respectively denote one fully connected layer, with nonshared weights. The feature representation of each modal sample is no longer a fixed value, but sampled from the distribution. Here, 𝜇 𝑚 𝑖 represents the features closer to the true representation under noisefree conditions, while 𝜎 𝑚 𝑖 represents the aleatoric uncertainty of the modality data. However, the gradients cannot be back-propagated after sampling, thus we employ the re-parameterization trick:EQUATIONHere, Eq. ( 3) can be interpreted as stable sample representation 𝜇 𝑚 𝑖 being perturbed by noise 𝜎 𝑚 𝑖 to obtain𝑧 𝑚 𝑖 . During training, the loss function of the ultimate task will mitigate the impact of this perturbation, leading to 𝜎 𝑚 𝑖 being learned to be very small. This results in 𝑧 𝑚 𝑖 being almost identical to 𝜇 𝑚 𝑖 , thus losing its ability to quantify uncertainty. Therefore, we adopt a prior method [12] and introduce a 𝐾𝐿 divergence regularization term: We constrain 𝑁 (𝜇 𝑚 𝑖 , 𝜎 𝑚2 𝑖 ) to be close to 𝑁 (0, 𝐼 ) so that 𝜎 𝑚2 𝑖 can accurately quantify the magnitude of uncertainty.EQUATIONIn the preceding section, we defined the concept of an "expert" and endowed it with the capability to capture uncertainty, allowing us to quantify the magnitude of uncertainty in noisy data based on the value of 𝜎 2 . Next, we will integrate it into the MoE framework. The MoE consists of two main components: the Expert and the Gate. The expert is responsible for processing each sample effectively, while the Gate determines which expert is best suited to handle a particular sample. We denote these two components as 𝐸 and 𝐺, respectively. Given the feature 𝑥 𝑚 𝑖 as input, and the gate selects the top k best experts, preserving the corresponding processing results 𝐸 for further steps. The algorithm for routing by the gate can be expressed as:EQUATIONwhere the output dimension of the Linear layer is equal to the number of experts. The 𝑇𝑂𝑃 𝑘 operation involves setting all values except the top k values to zero. We select the top k experts corresponding to the highest 𝑠𝑜 𝑓 𝑡𝑚𝑎𝑥 output scores, and use their scores as weights for the expert output results. When the total number of experts is 𝑁 , this process can be formalized as:EQUATIONHowever, during training, routing algorithms often tend to favor a few specific experts, rendering the remaining experts ineffective. This leads to the degradation of the entire framework into a static network, unable to dynamically adapt to different noise. To tackle this issue, we adopt a similar approach as in the Switch Transformer [23] , incorporating the following auxiliary loss:EQUATIONwhere 𝑅 𝑚 𝑗 represents the proportion of samples allocated to the expert 𝑗,EQUATIONMoreover, 𝐺 𝑚 𝑗 represents the proportion of weights allocated by the router to expert 𝑗,and it is defined as:EQUATIONThrough this regularization constraint, we enforce a similar number of samples allocated to each expert during training, and ensure that 𝐺 𝑚 yields weights of similar magnitudes for each expert.So far, we have obtained a dynamic network capable of capturing uncertainty. Next, we will empower the Gate the ability to choose experts based on uncertainty. Specifically, we use the magnitude of the 𝜎 𝑚2 𝑖 value to quantify the uncertainty in noisy data. A larger 𝜎 𝑚2 𝑖 value indicates greater uncertainty, while a smaller 𝜎 𝑚2 𝑖 indicates lesser uncertainty. During a forward pass, each expert generates a 𝜎 𝑚 𝑖 to express its uncertainty in processing the data. We aim for the Gate to select experts with lower uncertainty to handle the corresponding data. Therefore, we apply the following constraint called U-Loss:EQUATIONWith this constraint, during training, as the value of 𝜎 𝑚2 𝑗 increases, the Gate assigns a smaller weight 𝐺 𝑗 to the corresponding expert. Thus, when an expert's uncertainty is higher, its probability of being selected decreases.Through the aforementioned process, we obtain clear features ℎ 𝑚 𝑖 for each single modality. Combine features from different modalities by concatenating or adding them to create a joint feature ĥ, which can be utilized for downstream tasks. We employ a fully connected layer to predict the final result ŷand use Mean Squared Error loss as the loss function for this regression task,EQUATION)where 𝑛 represents the number of samples. Finally, we utilize 𝐿 𝑡𝑜𝑡𝑎𝑙 as the overall loss function,EQUATIONwhere 𝛼, 𝛽 and 𝜆 are trade-off hyper-parameter, as introduced in Section 4.2. 
experiments 4.1 datasets: 
 Dataset. To verify the effectiveness of our proposed model in face of different kinds of noise scenarios i.e., modality missing and modalities' noise, we conduct MSA experiments on three multimodal datasets, namely CMU-MOSI [31] and CMU-MOSEI [32] for modality missing, and MVSA-single [33] for modalities' noise. In addition to that, we also conduct an extensive experiment on NYU Depth V2 [34] In this dataset, each object is labeled with a class and an instance number (cup1, cup2, cup3, etc). Following previous work [9] , we also adopt the commonly used 9 out of the 27 scene categories and the remaining categories as "Others". Experiment Setup. In alignment with alternative methodologies, we employ FACET, COVAREP and BERT [35] as feature extractors for visual, audio and text modalities in both CMU-MOSI and CMU-MOSEI. For bimodal dataset, we utilize ResNet-152 as the feature extractors for RGB and Depth images, accompanied with BERT as the texual feature extractors. The hyperparameters we employed, denoted as 𝛼, 𝛽, 𝜆, were set to 1e-3, 1e-5, and 1e-3, respectively. Additionally, for the selection of the number of experts using the Top-k mechanism, we set the value of k to 3. Due to space constraints, further relevant details are provided in the supplementary materials. In short, We implemented all the experiments using Py-Torch on a RTX 3090 GPU with 24GB memory. We set the training batch size as 16 and train our model for 100 epochs. We run each experiments on the testing set and report the model's performance. 
results and analysis: 
 To demonstrate the superiority of our approach, we conducted comparisons with state-of-the-art methods on three widely-used multimodal emotion analysis datasets. This includes two tri-modal video datasets, CMU-MOSI and CMU-MOSEI. The methods compared include CubeMLP [2] , MHMF-BERT [27] , GCNet [11] , MSG-MBA [4] , ConFEDE [6] and DiCMoR [7] . In addition, it also includes a bi-modal text-image dataset, MVSA-Single. The methods compared include MMTM [28] , TMC [29] and QMF [9] . Furthermore, we tested the performance of our model in situations of modality missing and noisy data.We report the comparison results of our method with the current state-of-the-art methods on datasets CMU-MOSI, CMU-MOSEI, and MVSA-Single in Table 1 . From the table, it is evident that our method outperforms the current state-of-the-art methods in the majority of metrics, with a significant improvement. It is worth noting that our method achieves Acc2 and F1 metrics exceeding 86% on the CMU-MOSI dataset, pushing the model's performance to a new peak on this dataset. Additionally, our method also demonstrates outstanding performance on the CMU-MOSEI dataset, with Acc2 and F1 metrics surpassing the second closest by more than 1%.Our method also performs exceptionally well on the bimodal textimage classification dataset. It surpasses the current state-of-the-art methods by more than 1% in both Accuracy and F1 score metrics. At the same time, it is noticeable that our method does not outperform other methods on the Acc7 metric on the CMU-MOSEI dataset. We speculate that the high quality of the CMU-MOSEI dataset itself might have limited the advantage of our uncertainty routing method, resulting in slightly lagging performance compared to other methods.In addition to analyzing the aforementioned three widely used sentiment analysis datasets, we conducted further experiments. Specifically, we conducted additional validation on the scene recognition dataset NYU Depth V2. The sentiment analysis datasets we used encompass features from three modalities: text, audio, and vision. However, the real world consists of more than just these three modalities of data, hence we decided to conduct more experiments. The NYU Depth V2 dataset for scene recognition includes features from two modalities: RGB images and depth images. By utilizing inputs from both modalities, we aim to recognize the scenes depicted in the images. We report the experimental results in the right subtale of Table 1 . As can be observed from the graph, our approach still outperforms existing state-of-the-art methods even when including RGB images and depth images data. It is noteworthy that our method exceeds the state-of-the-art methods by 1% in accuracy, demonstrating a significant performance improvement. Additionally, our method performs remarkably well in terms of F1 Score. This proves the generalizability of our approach to applications beyond just text, audio, and visual modalities. In order to demonstrate the supremacy of our method in handling noisy data, we conducted experiments under scenarios of modality missing, Gaussian noise, Salt and Pepper noise, etc. The experimental results are reported in Tables 2 and 3 . From Table 2 , it can be observed that our method exhibits significant performance improvement compared to other methods under the modality missing condition. It is noteworthy that our method consistently outperforms the state-of-the-art methods by nearly 1% in all scenarios. Particularly, when considering only the visual modality, our method surpasses the second-best by nearly 3% across three metrics, demonstrating the advantage of our approach. Similarly, as only the acoustic modality is available, our EUAR outperforms the second-best by around 2%. In Table 3 , we report the performance of our method under the presence of Gaussian noise and Salt and Pepper noise. It is evident that our method achieves state-of-the-art performance under both noise conditions. Specifically, our method shows a noticeable lead, surpassing the second-best by 2% when the Salt and Pepper noise intensity is 5 and the Gaussian noise intensity is 10. Whether in the case of modality missing or Gaussian and Salt and Pepper noise, our method consistently demonstrates superior performance in noisy data scenarios.Ablation Study on Loss. In order to investigate the effectiveness of the loss function proposed in our routing strategy, we conducted thorough ablation experiments on the loss function on both CMU-MOSI and CMU-MOSEI. We performed ablation experiments on the uncertainty loss 𝐿 𝑘𝑙 and the uncertainty routing loss 𝐿 𝑢 separately, and the experimental results are shown in Table 4 . From the table, it can be observed that the performance of the model significantly decreases on all metrics when any of the loss is ablated. It is worth noting that when both losses are ablated, compared to the complete model, every metric of the model's result decreases dramatically, especially on CMU-MOSI by more than 4%. The 𝐿 𝑘𝑙 loss enables better quantification of uncertainty by experts, while the 𝐿 𝑢 loss helps the gate to route samples to the expert with lower uncertainty for processing. Especially, when we solely dissolve the KL-loss, our performance only declines marginally, by merely around 1% on both datasets. This further corroborates the validity of our introduction of MoE. The intrinsic dynamism of MoE confers an advantage in this task, beyond solely relying on more precise routing methods. However,the improvements we propose complement the inherent nature of MoE. Only when these two losses interact can the overall effectiveness of the model be optimized. Especially, when the text modality present in the combination, complete model possesses more evident performance. Contrastively, the complete model has more concentrated outcomes and reduced fluctuations, revealing its remarkable robustness and stability. Moreover, on NYU Depth v2, we also replaced the expert with a standard MLP and tested under four different noise conditions, as shown in right picture of Figure 3 . From the graph, it can be observed that our comprehensive approach significantly outperforms. It is worth noting that as the noise intensity increases, the advantage of our method becomes more apparent, demonstrating the excellence of our approach. Especially in the case of Gaussian noise with an intensity of 5, our method consistently outperforms the standard MLP in both average performance and stability. Ablation Study on Experts Number. To further explore the impact of different experts number on our proposed MoE module, we conduct ablation experiment on experts number, as shown in Fig. 4 . The numbers of experts range from 2 to 32, and it is clear to tell that as the number of expert increase, the model's performance present upward trend. The results indicate that with more experts, our model is able to exclude the noise in the samples more precisely, and more discriminatively extract the useful features that are benefit the final classification results. It is noteworthy that when we increase the number of experts from 2 to 32, the performance of our model on Acc2 improves by nearly 2%. This demonstrates the success of our proposed dynamic network strategy. However, it can be observed that the improvements in Acc2 metrics on CMU-MOSEI during the early part are not as significant when changing the number of experts. We analyzed that the CMU-MOSEI dataset has a larger volume of data, requiring a greater number of parameters for fitting. This is also why there is a significant increase in Acc2 during the latter part of the experiment. With an increase in the number of experts in our approach, there is an enhancement in our network's ability to perform finer-grained sentiment analysis. Visualizations of Joint Representations Furthermore, using the features that are projected into a 2D space by t-SNE is a straightforward way to exhibit learning joint representations of EUAR. Hence, we employ the t-SNE to visualize the learned joint representations of simple concatenation, MLP, and complete EUAR for a quantitative comparison. In detail, we randomly select joint features processed by our proposed EUAR model from the testing set of the CMU-MOSI dataset, and use three colors to represent their true sentimental labels.As illustrated in Fig 6 , the picture above is the visualization of complete modalities combination, while the figure below is the combination whose acoustic modality is unavailable. It is evident to observe that the features generated by the complete model are in compact and discriminative distribution. On the contrary, when MLP are utilized, the ablated model can separate the features in different classes to some degree, the final distribution is less compact and distinguishing, resulting in corrupted classification results at last. And the clusters are still closer than those generated by the complete EUAR, indicating less discriminability. Consequently, the t-SNE visualization results indicate that with the help of both MoE structure and proposed U-loss, effectively enables our EUAR model to resist noise in the modalities, making it more capable of learning a representative joint representations. Because of the limitation of the space, more missing modality combinations' t-SNE visualizations will be attached in the supplementary materials. Qualitative Analysis Additionally, for a qualitative study, we also illustrate several representative test cases from CMU-MOSI with different missing modalities situations. Specially, when the modality is unavailable in the combination, we use rectangles with red dash lines to denote. As shown in Fig 7,  we compare our proposed model with the recent counterpart DiCMoR [7] . Under different missing modalities circumstances, our EUAR method consistently produce correct classifications, while DiCMoR fails to fulfill the task accurately. It explicates that with the assistance of the MoE architecture and the novel routing algorithm we proposed, our model is capable of learning more stable and robust joint representations from deteriorated multimodal data and excluding the interference of noise that exists across the modalities, gaining noise-resistant performance with higher accuracy. 
conclusion: 
 In this paper, we proposed a novel multimodal sentiment analysis framework called Enhanced Experts with Uncertainty-Aware Routing (EUAR), which excelled at handling noise in multimodal data and dynamically adjusted the network based on different samples. Specifically, we introduced the MoE in multimodal sentiment analysis tasks to address varying noise levels in multimodal data. Particularly, we enhanced the functionality of experts to quantify uncertainty and extract clear unimodal features. Moreover, we devised novel routing strategies to train the model to route samples with different noise levels to corresponding experts with lower uncertainty for processing. Our approach outperformed existing state-of-the-art methods in multimodal sentiment analysis tasks and excelled in extended experiments with additional modalities. Furthermore, we conducted experiments under conditions of missing modalities and noisy data, demonstrating the superiority of our method in handling noisy data. For future work, we aim to delve deeper into methods for handling noise in multimodal data. 
 
 Appendix:  


Decision: This paper proposes a new method, Enhanced experts with Uncertainty-Aware Routing (EUAR), to better mitigate noisy date by dynamically adjusting the network for improving multimodal sentiment analysis. All the reviewers highlight the novelty of the approach and its effectiveness, advancing the state of the art. They all align towards acceptance.

Reviews: review: paper_summary: paper_summary: This paper applies the MoE (Mixture of Experts) framework in multimodal sentiment analysis. This involves dynamically adjusting the network to mitigate noise within individual modalities and across diverse samples. They develop a routing algorithm to bolster the capabilities of the individual experts within the network. This algorithm enables the network to assess the uncertainty associated with noise and leverage this information to guide the selection of experts with greater confidence for processing each sample. Their experiments have resulted in achieving state-of-the-art performance levels. Moreover, their method has showcased superior robustness in handling noisy conditions compared to existing techniques.
strength_weakness: strength_weakness: The authors introduce a pioneering approach by formulating a MoE framework in MSA, considering multiple modalities. The proposed framework presents a novel method to enhance the robustness capabilities of MSA models. The empirical results demonstrate the superior performance of the proposed method.
limitations: limitations: 1. Traditional methods such as TFN are missing as baselines, and classic MSA methods like Self-MM, MAG-BERT, and MISA are also absent.
2. Additionally, there is another piece of work addressing the generalization and robustness issues in MSA. The paper "General Debiasing for Multimodal Sentiment Analysis" published at ACM MM 2023 should be discussed to highlight the differences in approach compared to this work.
paper_summary: paper_summary: This paper introduces the Enhanced experts with Uncertainty-Aware Routing (EUAR) method to mitigate the impact of noisy data on MSA. Leveraging a Mixture of Experts approach, EUAR dynamically adjusts the network based on data uncertainty, refining features and routing mechanisms. Through the proposed U-loss, the network routes samples to experts with lower uncertainty, yielding clearer features. Experimental results, including on noisy datasets, showcase EUAR's superiority, establishing new benchmarks in multimodal sentiment analysis.
strength_weakness: strength_weakness: 1. The motivation for this study is relatively novel within the field, addressing the pressing need to mitigate the impact of noisy data on multimodal sentiment analysis. Data sourced from social media platforms often contain significant levels of noise, presenting a unique challenge for accurate analysis. 

2. The manuscript is commendably readable and easy to follow, facilitating understanding and comprehension of the presented research. 

3. The proposed method achieves state-of-the-art (SOTA) performance on selected datasets, indicating its effectiveness and competitiveness in the field of multimodal sentiment analysis. Additionally, the experiments conducted are comprehensive, covering various aspects necessary for thorough validation.
limitations: limitations: 1. In line 339, the phrase "being perturbed by noise 𝜎 to obtain𝑧" should include a space between "obtain" and "𝑧." Manuscripts submitted to top-tier conferences should avoid such low-level errors to maintain a high standard of professionalism and accuracy. Attention to detail in formatting and presentation is crucial for ensuring the credibility and quality of academic research.

2. This study places excessive emphasis on capturing data uncertainty, while insufficiently addressing the extraction of sentiment clues from different modalities. This focus detracts from the task-specific objectives of multimodal sentiment analysis, where the integration and interpretation of emotional cues across various data types are crucial. A more balanced approach that equally prioritizes sentiment extraction from all modalities would enhance the method's relevance and effectiveness for the intended task.


3. While the design of the algorithms is reasonable and effective, the presentation of the methodology may give readers the impression that it primarily addresses noise reduction in multimodal scenarios rather than focusing on multimodal sentiment analysis. Clarifying how the proposed method specifically enhances sentiment analysis, beyond merely mitigating noise, would strengthen the relevance and impact of the research within its intended context.

4. The concept of "expert" in this study is too abstract. The paper lacks clarity regarding the specific types of sub-models or neural networks chosen as experts within the framework. Providing detailed information about the architecture and characteristics of these experts would enhance the transparency and comprehensibility of the proposed method.

5. The overall loss function includes too many hyperparameters, making it difficult to determine whether all these components are beneficial for the task.
paper_summary: paper_summary: This paper focuses on multimodal sentiment analysis, and proposes a EUAR model, i.e. Enhanced experts with Uncertainty-Aware Routing, to address the influence of noisy data on multimodal sentiment analysis. EUAR can capture uncertainty by quantifying noise in samples to obtain clearer features and utilize this uncertainty to guide the routing algorithm according to different noise scenarios. This paper shows the advantages of EUAR comparing with related work by experiments.
strength_weakness: strength_weakness: 1. This paper proposes Uncertainty-Aware Routing, which introduces uncertainty to guide the routing algorithm. This is an active learning idea, and it should be novel for sentiment analysis.
2. The experiment setup is described clearly by this paper, and the experimental process is complete and includes comparison with other work, selection of parameters, ablation, and so on.
3. This paper makes full use of multimedia technique to show the experimental results and gives detail analysis.
limitations: limitations: About the details and process of EUAR model, the description of the model in this paper is not very clear, which greatly affects the readability and understandability of the paper. Main problems are following.
1. According to Figure 1, this paper classifies noise data into modal-wise noise and sample-wise. However, what is the meaning and function of this classification? How does EUAR model treat these two types of noise differently? 
2. As the main framework, figure 2 is not clear enough, or does not have a clear correspondence with the paper text. In the text, visual input can be extracted audio, visual, and text feature, what for text input? In Figure 2, for Enhanced Experts module, only the process of text can be seen. The paper labels (Textual/Visual) in the figure, but just replace T with V? In addition, Figure 2 should be more detailed, and some of the variables in the text should be marked more in the figure.
3. This paper classifies experiment datasets into modality missing and modality noise. On the one hand, how does this classification relate to the previous modal-wise noise and sample-wise? On the other hand, CMU-MUSI, CMU-MUSEI, and MVSA-single are multimodal sentiment analysis datasets, how does this paper process them to be suitable for modality missing and modality noise experiments? Do the 3 datasets themselves have modality missing and modality noise problems? If yes, the paper should give the statistical information of 3 datasets.