# Advancements in EEG Emotion Recognition: Leveraging Multi-Modal Database Integration
## 3 Multi-Modal System EEG with Facial Expressions--Proposed Model
### Fusion of Networks' Results  
The difficult problem of emotion recognition is greatly improved by combining data from multiple modalities. It has recently been demonstrated that combining facial expression with EEG-based CNN outputs is a potential method for emotion identification. When facial expression-based CNNs like DeepFace are combined with the output of EEG-based networks that capture brain activity, a complete and powerful emotion identification system may be created. This section looks at the results of the fusion between the two neural networks outputs and its effect on the overall system efficiency. It also goes over potential tactics and guidelines for this integration.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Layer (Type)** & **Output** & **Number of Param** \\ \hline
2D Convolutional & (None, 63, 63, 10) & 130 \\ Activation & (None, 63, 63, 10) & 0 \\
2D Max Pooling & (None, 12, 12, 10) & 0 \\
2D Convolutional & (None, 11, 11, 10) & 410 \\
2D Max Pooling & (None, 2, 2, 10) & 0 \\ Activation & (None, 2, 2, 10) & 0 \\ Flatten & (None, 40) & 0 \\ Dense & (None, 10) & 410 \\ Dense & (None, 2) & 22 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Valence CNN layers structure.  
The identification of emotional states, such as responses to certain stimuli or mental processes, may be preformed using EEG data. However, DeepFace CNN is made to use cameras to recognize facial expressions and provide accurate visual cues about emotional states that are observable from the outside. Enhancement of the efficiency and accuracy of the emotion identification system may be achieved via the merging of these two modalities. Data from EEG-based networks and DeepFace CNN output may be combined using a variety of fusion techniques [57, 58, 59]. Here are some possible methods for this fusion as examples:  
* **Feature-Level Fusion**: Combines features extracted from both EEG and facial expression data before classification.
* **Decision-Level Fusion**: Independent decisions from each modality are combined to reach a final classification.
* **Late Fusion**: Integrating the models themselves, such as through ensemble methods or hybrid architectures.
* **Early Fusion**: Combining the input data from the DeepFace and EEG-based CNNs and sending them to a single CNN for collaborative feature learning.  
In this study, we combine the outputs of DeepFace CNN with an EEG-based CNN, using a decision-level fusion technique to improve emotion recognition. By taking use of the unique advantages of each modality, this approach seeks to increase the accuracy and scope of emotion detection systems [60]. The main and key factor of this integration is the use of the neural networks' confidence scores (CS) for each network's output. The confidence score describes how confidently an input is classified by the model into a certain class or output. In the classification process, CNNs often give a CS, or the likelihood that the input belongs to a class. Typically, the class with the highest CS is selected for the final output. The softmax activation function is typically used to transform the output scores of the network into a probability distribution.  
Using this method to combine the networks' outputs is powerful for creating an emotion recognition system that is more accurate and reliable. Confidence scores play a critical role in determining if more validation or human intervention is necessary, particularly in critical applications like autonomous driving or medical diagnostics where the model's decisions have far-reaching consequences. Moreover, CSs are essential to the decision-making process and overall performance evaluation since they help users evaluate and trust CNN's outputs, as well as comprehend the behavior of the model. The CS score for the valence emotional axis, which summarizes the many combinations that may be made for emotion identification, is the basis for the system's outcomes, as indicated in Table 4.