# Reducing Videoconferencing Fatigue through Facial Emotion Recognition
## 2 Related Work
### Facial Emotion Recognition  
Facial emotion recognition (FER) is a stream of work in which researchers in the area of computer vision, affective computing, human-computer interaction, and human behavior deal with the prediction of emotions using facial expressions in images or videos [15]. FER literature can be divided into two groups according to whether the features are handcrafted or automatically generated through the output of a deep neural network [6]. Furthermore, FER research can be distinguished according to whether the underlying emotional model is based on discrete emotional states [16] or on continuous dimensions, such as valence and arousal [17]. In the former, researchers share a consistent notion of emotions as discrete states, although different determinations exist with respect to what these basic emotions are. For example, [PERSON] and [PERSON] [16] identified the five basic emotions as happiness, anger, disgust, sadness, and fear/surprise. [PERSON] [18] defines play, panic, fear, rage, seeking, lust, and care as basic emotions. In the continuous dimension model, emotions are described by two or three dimensions containing valence or pleasant as one dimension and arousal or activation as the other dimension [19]. Considering that we leverage discrete emotional states in our work as well as the fact that relatively few studies develop and evaluate algorithms for the continuous dimension model [19], especially for the handcrafted feature generation process, we will only focus on the discrete emotional model type that distinguishes between handcrafted and deep learning based approaches. FER approaches that use handcrafted features are usually deployed in three steps: face and facial component detection, feature extraction, and expression classification [6]. In the first step, faces and facial components (e.g., eyes and mouth) are detected in an input image. In the second step, various spatial and temporal features such as Histogram of Gradients (HoG), Local Binary Pattern (LBP), or Gabor Filters are extract from the facial components. Finally, machine learning algorithms such as Support Vector Machine (SVM) or Random Forests use the extracted features to recognize emotional states. Researchers have shown that manually extracting features can lead to accurate results [20; 21; 22; 23] and practitioners use such approaches because, compared to deep learning based methods, manually extracting features requires much less computational resources [6]. However, due to the rise in the size and variety of data sets and thanks to recent developments in deep learning, deep neural networks have been the most appropriate technique in all computer vision tasks including FER [15]. Many researchers showed the superiority of deep learning algorithms, with CNNS, RNNS, and LSTM models in particular, over handcrafted approaches [24; 25; 26; 27; 28; 29; 30; 31; 32; 33]. The main advantage of neural networks is their ability to enable \"end-to-end\" learning, whereby features are learned automatically from the input images [6]. One of the most deployed neural networks in the context of facial emotion recognition is the CNN [6], which is a special kind of neural network for processing images by using a convolutional operation [34]. The advantage of such a neural network is that it can take into account spatial information that is location-based information. However, CNNs \"cannot reflect temporal variations in the facial components\" [6] (p. 8) and thus, more recently, RNNs or LSTM models in particular, have been combined with CNNs to capture not only the spatial information but also the temporal features [7; 26; 30; 31; 32; 33]. In our study, we use a deep learning based method, a CNN, to recognize a subset of the [PERSON] model's emotions which are the following: anger, fear, happiness, sadness, and surprise. Furthermore, we augmented the emotions with the neutral expression as a state of control for the recognition results on emotions, similar to [PERSON] et al. [35]. Moreoever, as suggested by [PERSON] et al.