# Advancements in EEG Emotion Recognition: Leveraging Multi-Modal Database Integration
## 4 Results and Discussion  
The final step is the fusion between the two results from the two separate neural networks reporting human emotions using facial expressions and EEG heat maps. The fusion between networks' outputs could be performed in different ways. In this study, a decision-level fusion technique was used based on the confidence score. As previously stated, and illustrated in Table 4, the confidence score threshold was established at 0.7. This number was derived by a through a statistical study designed to resolve instances of inaccurate categorization. The decision process involved manually adjusting the confidence \begin{table}
\begin{tabular}{l l l l} \hline \hline
**EEG-Based CNN** & **FE-Based CNN Result** & **Valence** & **Output** \\
**Result** & & **Network CS** & \\ \hline Happy/Sad & Not Detected & Any & EEG result \\ Happy/Sad & Happy/Sad & Any & EEG result \\ Happy/Sad & Sad/Happy & \textgreater{}0.7 & EEG result \\ Happy/Sad & Any & \textless{}0.7 & Result with the \\  & & & highest CS \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Valence emotion classifications possible combinations and reported emotional state. score threshold. Subsequently, we computed the percentage of accurate results produced by the neural network when confidence scores exceeded the selected threshold, alongside the percentage of erroneous results when confidence scores fell below it. This assessment was conducted relative to the total number of attempts above and below the specified threshold, respectively. The study found that the vast majority of wrong classifications happened when the confidence score went below 0.7. In light of these findings, this threshold was carefully selected as the ideal spot for executing the fusion process. This approach guarantees a more dependable and accurate classification process by reducing the chance of mistakes associated with lower confidence levels, increasing the methodology's overall efficiency by depending on facial expressions' CNN results. The fusion technique depends on separately training EEG- and facial expression-based CNN models and combining their outputs using a weighted sum or majority voting to make the final emotion classification decision. The application of this mathematical expression involves considering the result from the EEG-based CNN when facial recognition by the camera yields negative results, when both CNNs have the same output, or when they have different emotion classes but the EEG-based CNN has a confidence score greater than 70%. The final situation is when each CNN reports a different emotional class and the EEG-based CNN has a confidence score less than or equal to 70%, in this case, the emotion reported with a higher confidence score will be considered. The subsequent amalgamation of outputs from the two aforementioned CNNs is manifested as an enhancement in the overall efficiency of emotional classification. The fusion system mainly depends on the EEG signals to avoid faking facial expressions or lighting and camera position misleading problems. As a result of applying the fusion equation to the results achieved by the two CNNs, the efficiency was improved from 87.43% to 91.21% by applying the algorithm on 3353 random images out of database samples previously labeled by the participants as happy or sad. Obviously, we can see that the multi-modal system depending on two different inputs enhances the overall system's accuracy as well as generalizes the system's usage. The multimodal emotion recognition system finds applications in affective computing, where it can be used to create emotionally intelligent systems that respond to users' emotional states. The system can enhance human-computer interaction by enabling devices and applications to adapt to users' emotions, leading to more personalized and engaging experiences. In the context of mental health, our system can aid in monitoring and understanding emotional states, potentially assisting therapists and individuals in managing mental well-being.