# Emotion Recognition by Correlating Facial Expressions and EEG Analysis
###### Abstract  
Emotion recognition is a fundamental task that any affective computing system must perform to adapt to the user's current mood. The analysis of electroencephalography signals has gained notoriety in studying human emotions because of its non-invasive nature. This paper presents a two-stage deep learning model to recognize emotional states by correlating facial expressions and brain signals. Most of the works related to the analysis of emotional states are based on analyzing large segments of signals, generally as long as the evoked potential lasts, which could cause many other phenomena to be involved in the recognition process. Unlike with other phenomena, such as epilepsy, there is no clearly defined marker of when an event begins or ends. The novelty of the proposed model resides in the use of facial expressions as markers to improve the recognition process. This work uses a facial emotion recognition technique (FER) to create identifiers each time an emotional response is detected and uses them to extract segments of electroencephalography (EEG) records that a priori will be considered relevant for the analysis. The proposed model was tested on the DEAP dataset.  
affective computing; EEG; emotions; FER; machine learning; neural networks Article  
Emotion Recognition by Correlating Facial Expressions and EEG Analysis  
1  
analyze cognitive phenomena, such as emotions, there is no explicit reference of when an emotional stimulus begins or ends, and the full spectrum of the signal is analyzed, assuming that there must be a trait within a period that characterizes the emotion. However, this assumes that the stimulus produced is expected and that there are no different responses throughout the experimental process, which is not crystal clear.  
We propose to use facial expressions as time markers that dictate which moments in brain signals' registration we should be focused on, and thus generate a more robust and accurate methodology. The structure of this proposal is shown in Figure 1. The first stage is to analyze the video records using a facial emotion recognition (FER) technique to obtain markers that will tell us which and when an emotion occurred. Later this information will be used to extract fragments of the signals from an EEG record (two seconds before the identifier appears) associated with these markers and create a collection of data that, a priori, can be categorized as linked to an emotional process. The information obtained will be subsequently processed using digital signal processing techniques and a recurrent neural network training phase.  
Finally, we use the structure presented in [8] to summarize the highlights of this article:  
* A methodology is proposed that allows the identification of the analysis space for detecting emotions through EEG analysis and thus increasing the certainty that a behavior linked to emotion is being recognized.
* A boosting methodology is used that uses two neural networks that operate together to strengthen the result.
* ESL optimized networks are used to reduce the effects of noise.
* An electrode reduction technique based on Brodmann's regions is used.
* The DEAP database is used as a reference point, and a recognition rate of 88.2% \(\pm\) 0.23 is obtained for the FER and 89.6% \(\pm\) 0.109 for the EEG analysis.