# HyFusER: Hybrid Multimodal Transformer for Emotion Recognition Using Dual Cross Modal Attention
## 4 Hybrid Multimodal Transformer for Emotion Recognition
### Hybrid Multimodal Transformer for Emotion Recognition Model
#### 4.3.3 Hyperparameter Settings  
The proposed multimodal emotion recognition model was trained with a well-structured configuration to ensure robust performance and strong generalization capabilities. The Adam optimizer, with a learning rate of 0.0001 chosen through empirical evaluation, balanced convergence speed and model stability. A batch size of 32 ensured efficient training while maintaining the ability to capture representative gradients for parameter updates. The model was trained for a maximum of 30 epochs, with early stopping applied to prevent overfitting by halting training when validation accuracy failed to improve for three consecutive epochs. This approach reduced unnecessary computations while preserving generalization performance. Additionally, a ModelCheckpoint mechanism saved the model weights corresponding to the best validation accuracy, ensuring the optimal model was preserved for subsequent evaluations. Activation functions played a pivotal role in capturing the nonlinear relationships inherent in multimodal data. The ReLU activation function, applied within the feed-forward layers of the transformer encoder and CrossModalAttention components, introduced nonlinearity and facilitated the learning of complex patterns. In the output layer, a softmax activation function computed class probabilities for the seven emotion categories. These probabilities from both modalities were combined through an ensemble averaging approach, leveraging complementary information to produce the final emotion recognition outputs.  
To stabilize training and mitigate overfitting, regularization techniques were integrated. Layer normalization, with an epsilon value of 0.000001, was applied after attention and feed-forward computations within the transformer encoder, ensuring numerical stability and consistent activation magnitudes. This contributed to improved training dynamics and performance reliability. The Sparse Categorical Crossentropy loss function was employed, as the emotion labels were represented as integers rather than one-hot encodings, aligning effectively with the dataset's multiclass classification structure. Architecturally, each modality--audio and text--was processed through three transformer encoder layers, enabling the extraction and refinement of hierarchical feature representations. A single attention head in each encoder simplified the model design while maintaining the capacity to capture essential cross modal interactions. The feed-forward network dimension was set to 768, providing sufficient capacity to represent rich multimodal features without incurring excessive computational overhead.  
In summary, the proposed training configuration and architectural design collectively optimized the model's ability to recognize emotions from multimodal inputs. The thoughtful selection of hyperparameters, combined with robust regularization and optimization strategies, ensured the model achieved both efficiency and state-of-the-art performance. These results underscore the effectiveness of the proposed approach and provide a strong foundation for future extensions and real-world applications in multimodal emotion recognition. The detailed configuration is summarized in Table 3.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Input Shape & (126, 768) \\ Feed-Forward Dimension & 768 \\ Number of Layers & 3 \\ Number of Attention Heads & 1 \\ Layer Normalization Epsilon & 0.000001 \\ Activation Function & ReLU \\ Optimizer & Adam (Learning Rate 0.0001) \\ Early Stopping Monitor & Val Sparse Categorical Accuracy \\ Early Stopping Patience & 3 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Hyperparameter details.