# Enhancing Multimodal Emotion Recognition through Attention Mechanisms in BERT and CNN Architectures
## 4 Results and Discussion
### Discussion  
In this study, we introduced a multimodal emotion-recognition approach that leverages both the speech and text modalities to better understand emotional states. From the  
Figure 8: Visualization of performances on the MELD dataset [11; 42; 43]. \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Research**} & \multirow{2}{*}{**Modality**} & \multicolumn{2}{c}{**Metrics**} \\ \cline{3-4}  & & **WA** & **WF1** \\ \hline [PERSON] et al. [11] & Speech + Text & 54.79 & 48.96 \\ \hline [PERSON] et al. [42] & Speech + Text & - & 65.8 \\ \hline [PERSON] et al. [43] & Speech + Text & - & 53.42 \\ \hline [PERSON] et al. [44] & Speech + Text & 65.59 & 64.50 \\ \hline Ours & Speech + Text & 66.81 & 66.12 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Comparison of the recognition performances on the MELD dataset. Figure 9: Training and validation Loss (**a**) and accuracy (**b**) performances on the MELD dataset. experiments and results presented, several key insights emerged that provide a comprehensive understanding of the potential and efficacy of the proposed model. Our multimodal emotion-recognition system is designed with an advanced architecture that integrates convolutional neural networks (CNNs) for processing mel spectrograms in the speech module and a pre-trained BERT model for the text module. This approach provides a significant advantage by extracting crucial temporal, frequency, and semantic features essential for a nuanced understanding of emotional expressions. The CNNs in the speech module utilize specific kernel sizes to capture a broad range of acoustic properties effectively, while the 12-layer bidirectional structure of BERT analyzes textual context from both directions, ensuring deep semantic comprehension. Further enhancing our model, hierarchical feature processing is employed within both modules. In the speech module, the layer-by-layer refinement of features through increasing channel numbers allows for the detailed extraction of acoustic signals. Similarly, in the text module, features processed by BERT are dimensionally reduced and contextually enriched through subsequent layers, including a fully connected layer and a Bi-GRU, optimizing the use of textual information for emotion recognition. The fusion of these processed features is managed by a dynamic attention-based mechanism, which adjusts the contributions of each modality based on their relevance to the current context. This not only improves the system's accuracy but also its adaptability, as the attention mechanism prioritizes modalities dynamically, enhancing performance in complex scenarios. Additionally, standardizing feature dimensions prior to fusion ensures equitable treatment of modalities, preventing bias and maintaining balance in attention allocation. Robust training protocols further strengthen the system, involving 300 epochs with strategic dropout and L2 regularization to prevent overfitting, thereby enhancing the model's generalizability and performance. This rigorous training approach maximizes the advantages offered by our sophisticated model design, ensuring that the system is capable of performing emotion recognition with exceptional accuracy and sensitivity to context. This architecture, coupled with strategic feature processing and dynamic modality fusion, positions our emotion-recognition system at the forefront of technological advancements, offering a powerful tool for accurately deciphering complex emotional states from multiple data sources. While the integration of these sophisticated models provides a robust framework for emotion recognition, it introduces substantial complexity and computational demands. This complexity can hinder real-time application and scalability, particularly in environments lacking extensive computational resources. Additionally, the reliance on large, annotated datasets for training these advanced models can limit their applicability in scenarios where such data are scarce.