# Combining Facial Expressions and Electroencephalography to Enhance Emotion Recognition
## 2 Materials and Methods
### Spontaneous Facial Expression Detection  
For predicting the emotion status based on facial expression, the following steps were taken. We first resampled the video to 4 Hz and used OpenCV's Viola & Jones face detector (frontal and profile) to find the face's position in each image frame [16]. The extracted faces were resized into a width and height of 48 pixels and fed directly to the multi-task CNN model to conduct a forward propagation. The outputs of the forward propagation were the status of valence and arousal.  
Figure 1: An overview of the workflow in this work.  
The training process and architecture of the multi-task CNN were as follows. The training process of the multi-task CNN was conducted in two steps. In the first step, supervised pre-training was performed. We discriminatively pre-trained the CNN using a large auxiliary data set (namely, fer2013) with image-level annotations [17]. In the second step, domain-specific fine tuning was performed. We continually conducted stochastic gradient descent (SGD) training for the CNN parameters using only faces extracted from the video with a relatively low learning rate (0.001).  
The illustration in Figure 2 shows the architecture of this network. Given the input \(48\times 48\) grayscale image patch, the first layer was a convolutional layer with 32 kernels of size \(3\times 3\times 1\). The second layer was a convolutional layer with 32 kernels of size \(3\times 3\times 32\). The third convolutional layer had 64 kernels of size \(3\times 3\times 32\), followed by \(2\times 2\) max pooling. The fourth layer was fully connected with 64 neurons. In all convolutional and fully connected layers, the rectified linear unit (ReLU) non-linearity approach was applied [18]. The network was subsequently split into two branches.  
The first branch learns the valence decision and contains two additional fully connected layers of sizes 64 and 1. The output is then input into a sigmoid function, and the cross-entropy loss \(L_{1}\) is minimized:  
\[L_{1}=-\sum_{i=1}^{m}\left(1-y_{1i}\log\dot{y_{1i}}+\dot{y_{1i}}\right)\log(1 -\dot{y_{1i}}) \tag{1}\]  
where \(y_{1i}\) represents the true label of valence of the \(k\)th sample, \(\dot{y_{1i}}\) represents the sigmoid output of the \(i\)th sample, and \(m\) represents the size of the training sample.  
The second branch learned the arousal decision and contained two additional fully connected layers of sizes 64 and 1. The output was fed to a sigmoid function, and we again minimized the cross-entropy loss \(L_{2}\):  
\[L_{2}=-\sum_{i=1}^{m}\left(1-y_{2i}\log\dot{y_{2i}}+\dot{y_{2i}}\right)\log(1 -\dot{y_{2i}}), \tag{2}\]  
where \(y_{2i}\) represents the true label of arousal in the \(i\)th sample, \(\dot{y_{2i}}\) represents the sigmoid output of the \(i\)th sample, and \(m\) represents the size of the training sample.  
When multi-task learning was performed, we minimized the following linear combination of losses:  
\[L=\sum_{p=1}^{2}\alpha_{p}L_{i}, \tag{3}\]  
Figure 2: The CNN network for multitask classification; “DO” indicates that the layers drop out [15]. Note that we apply padding in each layer except the second layer. Therefore, excluding the second layer, the output of each layer is of the same size as the previous layer.  
where \(\alpha_{p}\) are linear weights currently set to 1. Note that if we set the second weights to 0, we return to the conventional approach of single-task learning.  
After training, we could obtain the predicted result from the output of the network \(S_{face}\), as shown in (4), where \(r_{face}\) represents the result of valence or arousal based on facial expression:  
\[r_{face}=\left\{\begin{array}{ll}high&S_{face}\geq 0.5\\ low&S_{face}<0.5\end{array}\right. \tag{4}\]