# Reducing Videoconferencing Fatigue through Facial Emotion Recognition
## 3 Data and Methods
### Facial Emotion Recognition  
In order to identify participant's emotions, we used convolutional neural networks (CNNs) to employ facial emotion recognition (FER). In particular, we focused on six different emotions, inspired by [PERSON] and [PERSON] [16], namely: anger, fear, happiness, neutral, sadness, and surprise. The classifier was trained on a variety of available FER data sets where each face was labelled manually with a corresponding emotion. Specifically, we collected the following data sets:  
* 5405 images from AffectNet [19], which we labelled manually;
* 31,051 images from FERPlus ([[https://github.com/microsoft/FERPlus](https://github.com/microsoft/FERPlus)]([https://github.com/microsoft/FERPlus](https://github.com/microsoft/FERPlus)), accessed on 10 May 2021);
* 250 images from the Extended Cohn-Kanade Data set (CK+) [12];
* 184 images from the Japanese Female Facial Expressions (JAFFE) database [13];
* 522 images from BU-3 DFE [14];
* 3455 images from FFQH ([[https://github.com/NVlabs/ffhq-dataset](https://github.com/NVlabs/ffhq-dataset)]([https://github.com/NVlabs/ffhq-dataset](https://github.com/NVlabs/ffhq-dataset)), accessed on 10 May 2021), which we labelled manually.  
The images from AffectNet and FFQH were labelled by three researchers. All researchers received the same images and had to choose one of the following classes for each image: anger, fear, surprise, sadness, neutral, happiness, or unknown. The only images that were considered are those where two of the three researchers agreed on the same emotional state and ignoring images where the majority vote was on the unknown class.  
Finally, we combined all images, which resulted in a large and heterogeneous FER data set containing 40,867 images (with 8.58% of anger, 3.26% of fear, 13.81% of surprise, 11.45% of sadness, 33.13% of neutral, and 29.77% happiness). Table 1 presents the distribution of emotional states along the six data sets. Note that we used 80% of the images for  
Figure 1: Overview of the data pre-processing steps. Each recording is divided into eight videos and each contains a different presentation. We then extracted one image per second for each presentation video. Finally, we extracted all faces from a single image and classified them into presenter, supervisor, and audience.  
training, 10% for testing, and 10% for validation. The validation set was used to estimate the prediction error for model selection. That is, the training of the model was terminated prematurely (before the maximum epoch) once the performance on the validation set became worse.  
Regarding the deep models, we considered several widely-used convolutional neural network (CNN) architectures including VGG [11] and Xception [44]. We created these models from scratch by utilizing Keras. We adopted the cross-entropy criterion as the loss function which is minimized using the Adam optimizer with a learning rate of 0.025. We trained each model up to 100 epochs. After training, we evaluated each model on the same testing set of the heterogeneous FER data set and chose the model with the highest performance in terms of accuracy. Subsequently, the selected model was used to predict the emotions for all faces that we had previously recorded in the 41 presentations.