# Enhancing Multimodal Emotion Recognition through Attention Mechanisms in BERT and CNN Architectures
## 3 Proposed Multimodal Emotion-Recognition Approach
### Fusion Module and Classification  
When classifying sentiments, not every modality contributes relevance or significance to the same degree. Different modalities such as text, speech, and facial expressions offer unique aspects of emotional insight. However, their contributions to sentiment classification vary significantly. Some may have a more pronounced influence, whereas others may make only subtle contributions. This discrepancy arises because of the inherent differences in the types of information that these modalities encapsulate, as well as the varying capacities with which they express emotional cues. Therefore, when designing a multimodal system for sentiment classification, accounting for the diverse significance levels of the different modalities and strategically balancing their integration to achieve optimal performance is crucial. In the fusion module of our proposed multimodal emotion-recognition approach, we incorporated an attention-based fusion mechanism, as outlined in existing research [33]. This sophisticated approach allowed our system to assign varying degrees of importance to different modalities during the fusion process. The intention is to focus more heavily on the modalities that are considered most significant for a given context or dataset. This strategic prioritization helps refine the system output, enabling it to yield more accurate and contextually appropriate results in sentiment classification tasks. The research outlined in [33] incorporated three distinct modalities--audio, visual, and textual--to generate an attention score for each modality and determine its relevance to the final output. In contrast, our approach operates on a bimodal system, focusing solely on speech and text modalities. Despite this, our model effectively assigned attention scores to each of these two modalities, allowing for the dynamic allocation of significance based on their inherent contributions to the emotion-recognition task. Before inputting the feature vectors from the speech and text modalities into the attention network, their dimensions were uniform. This was accomplished using an FC layer of size n. By implementing this layer, we could effectively transform feature vectors into equal dimensions, thereby ensuring that each modality was represented equally when processed in the attention network. This step was vital for maintaining fairness and balance in the allocation of attention weights between the two modalities. Consider \(\text{F}=[\text{F}_{\text{s}},\text{F}_{\text{t}}]\) as the standardized feature set, where \" \(\text{F}_{\text{s}}\)\" represents the acoustic features, and \"\(\text{F}_{\text{t}}\)\" denotes the textual features. Here, the dimensionality of each feature set has been equalized to size \"n\", resulting in \"F\" belonging to the dimensional space \(\text{F}\in\mathbb{R}^{n\times 2}\). Optimal performance was achieved when the value o of n was set to 250. The attention weight vector, denoted as \(v_{at\_fusion}\), and the fused multimodal feature vector, referred to as \(M_{f}\), were calculated using the following procedures:  
\[L_{M_{f}}=tanh\Big{(}P_{M_{f}}\cdot\text{F}\Big{)} \tag{1}\]  
\[v_{at\_fusion}=softmax\Big{(}w_{M_{f}}^{\top}\cdot L_{M_{f}}\Big{)} \tag{2}\]  
\[M_{f}=\text{F}\cdot v_{at\_fusion}^{\top} \tag{3}\]  
where \(P_{M_{f}}\in\mathbb{R}^{n\times n}\), \(w_{M_{f}}\in\mathbb{R}^{n}\), \(v_{at\_fusion}^{\top}\in\mathbb{R}^{2}\), and \(M_{f}\in\mathbb{R}^{n}\). The resulting output, denoted as \(M_{f}\), signified the combined multimodal feature vector. This composite representation was then processed through a fully connected layer, which acted as an intermediary step to consolidate the high-dimensional multimodal features to be more manageable. Subsequently, a softmax layer was employed to finalize the speech emotion classification task. The softmax layer worked by outputting a probability distribution over predefined emotional categories, thus determining the most likely emotion to be present in the speech.