Deep facial emotion recognition in video using eigenframes

IET Image ProcessingVolume 14, Issue 14 p. 3536-3546 Research ArticleFree Access Deep facial emotion recognition in video using eigenframes Noushin Hajarolasvadi, Corresponding Author Noushin Hajarolasvadi noushin.hajarolasvadi@cc.emu.edu.tr orcid.org/0000-0002-3120-5370 Department of Electrical and Electronic Engineering, Eastern Mediterranean University, Mersin 10 TurkeySearch for more papers by this authorHasan Demirel, Hasan Demirel Department of Electrical and Electronic Engineering, Eastern Mediterranean University, Mersin 10 TurkeySearch for more papers by this author Noushin Hajarolasvadi, Corresponding Author Noushin Hajarolasvadi noushin.hajarolasvadi@cc.emu.edu.tr orcid.org/0000-0002-3120-5370 Department of Electrical and Electronic Engineering, Eastern Mediterranean University, Mersin 10 TurkeySearch for more papers by this authorHasan Demirel, Hasan Demirel Department of Electrical and Electronic Engineering, Eastern Mediterranean University, Mersin 10 TurkeySearch for more papers by this author First published: 14 October 2020 https://doi.org/10.1049/iet-ipr.2019.1566Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Recently, video-based facial emotion recognition (FER) has been an attractive topic in the computer vision society. However, processing several hundreds of frames for a single video of a particular emotion is not efficient. In this study, the authors propose a novel approach to obtain a representative set of frames for a video in the eigenspace domain. Principal component analysis (PCA) is applied to a single emotional video extracting the most significant eigenframes representing the temporal motion variance embedded in the video. Given that faces are segmented and normalised, the variance captured by PCA is attributed to the facial expression dynamics. The variation in the temporal domain is mapped to the eigenspace reducing the redundancy. The proposed approach is used to extract the input eigenframes. Later, VGG-16, ResNet50, and 2D and 3D CNN architectures called eigenFaceNet are trained on the RML, eNTERFACE′05, and AFEW 6.0 databases. The experimental results are superior to the state-of-the-art by 8 and 4% for RML, eNTERFACE′05 databases, respectively. The performance achievement is also coupled with a reduction in the computational time. 1 Introduction Recently, facial emotion recognition (FER) in the video has been a trending topic in the computer vision society. Two challenges in the video processing problems are a large number of extracted frames and the temporal redundancy. In summarising a video, there are two main concerns: first, maintaining the variability of the domain-related information, and second, decreasing the redundancy and computational time of processing for improved performance. With the promising results of convolutional neural networks (CNNs), the cost of tensor processing becomes an important issue. This issue becomes even more important considering video processing problems. In general, keyframe extraction is categorised into the motion-based analysis, shot boundary-based analysis, content-based analysis, and clustering-based analysis. These approaches have several disadvantages. First, they are expensive in terms of computational time and complexity. Second, they do not reduce the redundancy efficiently [1, 2]. In the motion-based analysis, the idea is tracking the facial expression changes through calculation and comparison of the optical flow for each frame. One disadvantage of this approach is considering only the local variation which results in redundant content [3]. In the shot boundary-based approach, shots of the original video sequence are first detected and then one or more keyframes are extracted from each shot [4]. Shot transition is detected either by pixel-based methods or histogram-based methods. Firstly, pixel-based methods are highly sensitive to the motion. As a result, they suit where detection of camera or object movements are required. On the other hand, although the computational cost of histogram-based methods [5] is less expensive, yet they capture only the global properties of a frame rather than both the local and the global information. Another disadvantage of this method is that similar histograms may be the output of the entirely different images. The content-based analysis combines various conditions like shot-based and motion-based criteria or colour and texture features [6]. In clustering approaches, the idea is to group frames with similar posture under different clusters [7]. The closest frames corresponding to the centroids of each cluster are selected as a keyframe. One drawback here is that the frame base feature extraction is essential. Despite the high accuracy obtained by pixel-to-pixel comparison, yet it is computationally exhaustive. Examples of such pre-processing are the computation of the entropy difference [8] or geometric features like distances between two successive frames. Another drawback in clustering methods includes random initialisation of centroids and convergence, which is not guaranteed. In this paper, we propose a novel approach that addresses three issues: capturing discriminative information, decreasing the time cost, and improving the performance. In this method, principal component analysis (PCA) is applied to the sequence of a single video extracting the most significant eigenframes representing temporal motion variance embedded in the video. The number of eigenvectors capturing sufficient variance determines the number of eigenframes be used. This dimensionality reduction gives the advantage of considering the variation of emotion within the whole facial expression video. Considering the FER problem, the sufficient variety in a single emotional video reflects the local changes in the temporal domain describing the facial expressions. The variation in the temporal domain is mapped to the eigenspace reducing the redundancy. Given that faces are segmented and normalised, the variance captured by the PCA is attributed to the facial expression dynamics. In the experimental procedures, two datasets are selected: Ryerson Multimedia Laboratory (RML) database [9], eNTERFACE′05 Audio–Visual Emotion database [10], and AFEW 6.0 [11]. Once the principal eigenframes with the cumulative variance of 90% are selected. VGG-16, ResNet-50 are fine-tuned by that set as input samples. Additionally, we propose 2D eigenFaceNet and 3D eigenFaceNet architectures considering the characteristics of the eigenframes. This architecture is employed to study the effect of large kernels, less number of layers, and training of CNNs from scratch using eigenframes. Assuming that each video is represented by k eigenframes, one class label is predicted for each. In the end, the majority voting is applied to the predicted class labels performing video-based expression recognition. Comparing to the state-of-the-art in the literature, the experimental results are superior by 8 and 4% for RML and eNTERFACE′05 database, respectively. The main contributions of this work are (a) Dimensionality reduction of a single facial expression video with hundreds of frames by mapping into eigenspace and reducing the computational time without loss of performance. (b) To the best of our knowledge, this is the first time eigenframes are extracted from a sequence of single emotional video to reflect facial expression dynamics in the temporal domain. This approach differs from other PCA-based methods which mainly focus on extracting eigenvectors from a set of labelled videos. (c) Evaluating eigenFaceNet as a 2D CNN attuned with the characteristics of eigenframes. The rest of the paper is organised as follows: in the proceeding section, related research is reviewed briefly. Then the proposed method is explained followed by the experimental results. Conclusion and future work are explained in the last section. 2 Related research Selecting the prominent frames in a video is a challenging task. Generally, keyframe extraction methods are computationally expensive. These methods can be categorised as motion-based analysis, shot boundary-based analysis, content-based analysis, and clustering-based analysis. Wang and Guan [9] used one keyframe to perform FER in the video. This keyframe is selected according to the highest speech amplitude. This selection is based on the idea that human facial expression peaks at a large voice amplitude. This approach is not applicable where unimodal emotion recognition based on visual modality is performing. In another recent study, Noroozi et al. [2] applied k-means clustering on a set of geometric features extracted from consecutive frames. Then, the closest frame to the centroid of each cluster is selected as the keyframe. Although this method provides high accuracy due to tracking motion through landmark points, yet it suffers from high computation time at the pre-processing stage. In a very similar study [12], k-means is applied to action units to select five keyframes. Also, in [13], Hajarolasvadi and Demirel applied the k-means clustering algorithm to acoustic features and then generated the corresponding spectrogram of the selected key audio segments. Another approach toward keyframe selection forms based on the assumption that the subject plays the emotion so that it starts from a neutral face, peaks at the considered emotion, and returns to the neutral face again. According to this assumption, Avots et al. [14] calculated the sum of absolute differences in a pixel-wise manner to generate a similarity score for frames. Then, they average these scores over the last ten frames, and only if the difference value for the current frame is greater than an empirical threshold (), the frame is selected as a keyframe. The drawback of this method is the varying threshold from one database to another. In addition, the onset-peak-offset assumption is not strictly true about the databases recorded in the wild. It is worth to note that some databases provide information about the onset and peak of the expression. For example, extended Cohn–Kanade dataset [15] is released so that the first frame be the neutral face and the video ends at the peak. Khorrami et al. [16] used four keyframes to represent each video of this dataset. To be more accurate, the first frame is selected as the neutral face and the last three frames as the peak of emotion. In a recent research study [17], Nguyen et al. introduced a model based on C3D CNN cascaded with deep belief networks where multimodal data used to perform emotion recognition. As of visual modality, they extracted all the frames and applied the Viola–Jones algorithm to detect the face region. Then using all frames they trained the 3D CNN on the visual data. More recently, Zhou et al. [18] explored emotion features in multimodal video classification systems by using attention mechanisms to highlight important emotional features. Other research works on video-based FER includes using all frames [19, 20] or selecting a random subset of the frames [21]. The former approach requires considerable computational time, especially when deep learning approaches are employed. The later does not support the selection of the frames with the maximum relevance to the target class and minimum relation with each other. Additionally, we would like to review the researches in the literature considering the main contribution of this paper, i.e. frame selection by mapping into eigenspace. For instance, a FER system is proposed based on eigenfaces by De et al. [22], where PCA is applied reducing the dimensionality of the eigenspace. Later, the test image is projected to the eigenspace, and the Euclidean distance between the test image and meaning of the eigenfaces of the training data is calculated. In this method, the PCA is applied to the whole training data and the database comprises single images and not sequence/video images. In our method, PCA is applied separately on the sequence of frames in one video. Belhumeur et al. [23] employed eigenfaces and Fisherfaces for emotion recognition. In the proposed method, PCA is applied to a set of N sample images where each image belongs to one of c classes. This differs from our method that PCA is applied to a sequence of frames in one video without assuming any class label for the frames. Jarlier et al. [24] used PCA decomposition of the thermal signal to show that thermal fluctuations are specific to the activated AUs. In their approach, PCA is employed for assessing temperature change in different face areas. Spatial PCA is applied to pixels and the temperature values to decide the relationships between the various temperature values measured for each pixel. Other recent research works acquired PCA for purposes like face recognition [25], dimensionality reduction of the feature space [26], and feature selection [27] which are different in nature of the application than frame selection by mapping into eigenspace. 3 Proposed method In the following section, first, eigenframe generation and selection are explained. Then, the classification models and their corresponding parameters and architectures are discussed. 3.1 eigenframe extraction PCA algorithm is used for generating the eigenframes. PCA is a technique widely used for dimensionality reduction in computer vision society and specifically in the face recognition analysis [23, 28, 29]. PCA or Karhunen–Loeve method chooses a dimensionality reduction by a linear projection that maximises the separation of all projected samples. Let be the set of N sample images in an n-dimensional space. Additionally, assume that a linear transformation maps the n-dimensional space into an m-dimensional feature space, where . Then, the new feature vectors are expressed as a linear transformation as follows: (1)where is an orthonormal matrix. We calculate a scatter matrix as follows: (2)where is the mean facial image. After that, linear transformation is applied to obtain scatter of the new feature vectors as . To maximise the determinant of the scatter matrix of the projected samples, projection is chosen as follows: (3)where is the ith n-dimensional eigenvector of . The first orthogonal dimension of this eigenspace captures the greatest amount of variance in the database whilst its last dimension captures the least amount of variance in the database. These eigenvectors have the same dimension as the original images. Hence, they are regarded as eigenframes. We applied PCA on the frame sequence of each video to select a representative set of eigenframes. In order to select the optimal subset, different approaches are proposed [30]: (a) Corresponding eigenvectors of non-zero eigenvalues are selected to create the optimal eigenspace. (b) The bottom most 40% of the eigenvectors is discarded. (c) It is assumed that the first three eigenvectors are affected by lighting conditions and thus, degrades classification performance. As a result, all eigenvectors except the first three are used. (d) The minimum number of eigenvectors which guarantee that the energy e is greater than a typical threshold is computed. Let be the accumulated energy of the ith eigenvector. Then it is calculated as , where k is the number of non-zero eigenvalues and shows the eigenvalue [31], (e) The stretch s of the ith eigenvector is defined as the ratio of the ith eigenvalues over the maximum eigenvalue and it is expressed as follows: [31]. A typical threshold for the stretching is 0.01. Most of the keyframe selection methods rely on the motion differences [2, 14]. Motion differences can be calculated as the amount of movement between different components of the face like lip and eyebrows. However, such methods require exhaustive numerical calculation. A more general method can be developed based on the amount of variance explained within data. In this study, we used the amount of variance explained by each of the principal components obtained from a single video. In fact, the percentage of variance explained by the principal components is considered to be greater than 90%. Variance in an emotional facial expression video corresponds to local changes of facial expressions in the temporal domain. This means the variance explains the motion differences correlated with the facial expression dynamics. The number of principal eigenframes (components) with the cumulative variance of 90% was 20, 25, and 17 for RML, eNTERFACE′05, and AFEW 6.0 databases, respectively. This idea is evaluated by visualising the class activation maps for the last layer of the models trained on the eigenframes. Fig. 1 shows the explained variance in percentage versus the number of the principal components for two random videos of each database with 175, 175, and 120 frames. The marked point shows the number of components where 90% of the data variation is captured. Fig. 1Open in figure viewerPowerPoint Data variation of one video from each database (a) RML database, (b) eNTERFACE′05 database, (c) AFEW database Several methods were employed to reduce the number of frames in one video from several hundred to a fewer amount. For comparison, we compared our method with two widely used methods in the literature. The first method is k-means algorithm used for keyframe selection by the authors in [2, 13]. We adopted the keyframe selection procedure proposed by Noroozi et al. [2] which is based on the k-means clustering. In this method, the authors extract 68 landmark points for each frame. Then, 60 geometric features including distances and angles are computed using the landmark points. K-means is applied to the geometric features to create k clusters based on the similarity of the frames. The closest frame to the centroid of each cluster is selected as a keyframe. We implemented this method for k equal to the same number as eigenframes. Another approach is a random selection of frames for each video [21]. In this study, we randomly select k frames from each video showing that the systematic selection of the frames outperforms the random selection. One problem of random selection is that the achieved set might be a set of entirely varied or entirely similar keyframes. To avoid fitting the model to such scenarios, We performed a random selection three times and the average accuracy of training over those sets is reported. K is chosen equal to the same number as eigenframes. Also, Fig. 2 illustrates the first five eigenframes obtained for a video in each emotion class of the RML database. The main characteristics of the relative class are retained in the eigenframes. It is important to highlight that PCA is used widely for face or identity recognition by applying PCA to the set of labelled videos. However, PCA is not used as a dimension reduction (frame selection) for a single video. Fig. 2Open in figure viewerPowerPoint Visualisation of eigenframes, each row shows the first five eigenframes of a random video within the labelled class 3.2 Classification The promising results of CNNs on computer vision problems and their growing application on different topics inspired us to use it as the classifier for evaluation and fair comparison with the state-of-the-art in the literature. We selected two pre-trained 2D CNN, namely VGG-16 [32] and ResNet-50 [33] and fine-tuned using the selected input samples obtained by different keyframe selection methods. Also, in a third experiment, we evaluate the performance of freshly trained CNNs using eigenframes. Following, the specific parameters of each experiment are explained. 3.3 VGG16 In 2014, the Oxford Visual Geometry Group won the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) by proposing the VGG-16 model. In this model, the input image is passed through a series of convolutional layers to apply small filters of size with stride 1. Some of the convolution layers followed by Max-pooling kernels of size and stride 2. This stack of convolutional layers is followed by three fully-connected (FC) layers. The final layer is a soft-max layer. All hidden layers use the non-linear rectification. To adjust the parameters of the pre-trained VGG-16 network, stochastic gradient descent (SGD) was used with momentum set to 0.9. Fine-tuning is performed by replacing the classification layer of the pre-trained VGG-16 with a soft-max layer that has the same number of units as the number of target classes (6). We initialised the network with the regular imagenet weights. Then, weights of the convolutional layers of the pre-trained network are fixed and only the three FC layers are fine-tuned. This preserves the generalisation of the earlier features of a pre-trained CNN, which are universally beneficial for different tasks. It is known that ending layers of the CNN extract the details more specifically. The batch size is set to 32. 3.4 ResNet-50 Similarly, residual network (ResNet50) was the winner in the ILSVRC challenge in 2015. He et al. [33] proposed residual nets which are eight times deeper than VGG nets, yet having lower complexity. Here, the convolutional layers mostly have filters of size . However, the layers have the same number of filters for the output feature map of the same size. The number of filters is doubled provided that the feature map size is halved. This preserves the time complexity per layer. Pooling is performed within the convolution layers. A global average pooling layer and an FC layer with soft-max function end the network. The total number of weighted layers is 34. Adjusting the parameters of the pre-trained ResNet-50 network is performed using SGD with the same parameters used for VGG. Also, weight initialisation, regularisation parameters are not changed. The same batch size as VGG-16 is used. 3.5 EigenFaceNet The block diagram of the proposed 2D and 3D CNN architecture is shown in Fig. 3. The spatial size of the kernels in 2D network is shown as , where H and W are the height and width dimensions, respectively. In case of 3D CNN, the size of the kernls for all layers is shown as , where D is the depth of the kernel. All kernels are set to . We refer to these networks as 2D eigenFaceNet and 3D eigenFaceNet in the rest of the paper. Fig. 3Open in figure viewerPowerPoint Block architecture of 2D and 3D versions of the eigenFaceNet (a) 2D eigenFaceNet, (b) 3D eigenFaceNet We employed SAVEE database [34] to search for an optimal CNN architecture. Starting from three layers, we try various square-shaped kernels of sizes 3, 5, and 7. Changing the size of kernels is applied homogeneously in all layers. Later, we increased the number of layers to 5 and 7, where various kernel sizes are assessed for each as well. The best CNN architecture was the three layer structure with kernels of size 5. Using such a network, we study the effect of shallow structure and large kernels on the improvement of accuracy. The learning rate in this experiment is set to 0.0001, the batch size is 32. We used a different optimiser than SGD for the fresh training of eigenFaceNets. In fact, it is trained from scratch using Adam optimiser with a batch size of 32. The initial learning rate is set to 0.0001 and decays exponentially to 0.00001. The total number of epochs is set to 400. The bias parameters are initialised to zero to have a fast training procedure by decreasing the number of learning parameters [16]. Weights are initialised from a uniform distribution. One dropout layer with a rate of 0.5 is used before the FC layer. This experiment is mainly performed to study the effect of training fresh 2D and 3D CNNs using eigenframes on the FER problem. Furthermore, we tried to observe if using large kernels improve learning using the eigenframes. In 2D CNN, the size of the input image is while this is reduced to for the 3D network due to memory limitation. Data augmentation is applied for training all three networks to avoid overfitting. Not all augmentation approaches could work over facial images and particularly on eigenframes. The data augmentation technique must be selected considering the dataset and the problem domain under discussion. Hence, the forms of data augmentation used in the experiments include only vertical and horizontal flips, scaling, and rotation by [16]. 3.6 Computation time analysis To evaluate the effect of using eigenframes as a representative subset for emotion recognition in video, we conducted the computation time analysis. The basic idea is to compare the computational time of the proposed method with other keyframe extraction methods. Also, the computation time for using all frames and all eigenframes is reported to show that the proposed method helps reduce the time for improved accuracy. Three types of temporal cost can be expressed. First is the time required for the pre-processing stage. Pre-processing time cost is defined as the time required for the extraction of the keyframe or the discriminant input samples using the corresponding method. The second computational time cost consists of fine-tuning/training for the networks. This time is known as offline or training time. The last temporal cost considered important in real-time problems is known as inference or online time, which costs during the evaluation of the model on unseen or test data. In 4.3 of this paper, we are studying all three types of time costs. 4 Experiments and analysis The main hypothesis of this paper is to improve video-based FER performance using a representative set of frames. Three principal experiments conducted by training VGG-16, ResNet-50, 2D eigenFaceNet, and 3D eigenFaceNet using the extracted eigenframes for each video. For all experiments, 90% of the database is used for training and the remaining 10% is used at the test phase. To overcome the overfitting problem and generalise the results, five fold-cross validation is applied to the train data. Separation of data to train and test for RML and eNTERFACE′05 is performed so that all videos of one subject across all categories either participate in the training procedure or the test. This undertakes the experiments to be subject independent. 4.1 Datasets We selected three databases, namely Ryerson Multimedia Laboratory (RML) and eNTERFACE′05. Both databases support audio and visual modalities and they include a wide range of ethnicity. 4.1.1 Ryerson Multimedia Laboratory RML presented this dataset consisted of eight subjects who acted six basic emotions: angry, disgust, fear, happiness, sadness, and surprise. 720 video samples recorded in a quiet environment with a bright, plain background. The videos recorded with a 30 F/s rate. Spoken languages of the actors cover a wide range from English and Mandarin to Urdu, Italian, and Persian. 4.1.2 eNTERFACE′05 The second dataset is eNTERFACE′05 provided by Martin et al. [10]. 42 subjects of various nationalities are asked to act the same emotion as the RML database. The spoken language is limited to English and 81% of the subjects were male. 210 video samples exist in each category. 4.2 AFEW 6.0 The Acted Facial Expression in the Wild (AFEW) database [11] is the main evaluation platform for the annual Emotion Recognition in the Wild Challenge (EmotiW) collected from movies. Samples are labelled with six basic expressions and neutral category and it includes 773 samples in the training set and 383 samples in the validation set. We used the validation set as of the test set and cross-validation is applied on the training set. Video samples with no detected face are discarded. Fig. 4 illustrates one random frame from a video in each database. The alignment is performed during frame extraction. Fig. 4Open in figure viewerPowerPoint Example of extracted frames for six prototypic emotions in RML, eNTERFACE′05, and AFEW 6.0 databases 4.3 Experiments We report the accuracy, loss, and area under the curve (AUC) of receiver operating characteristics obtained by fine-tuning the pre-trained VGG-16 and ResNet-50 as well as training the eigenFaceNets. It is important to note that the same data augmentation techniques are used in all experiments. In this study, fine-tuning and data augmentation is performed to boost performance and prevent overfitting. 4.3.1 Performance on RML database The discriminative ability of each CNN is evaluated on the RML database. Table 1 compares the accuracy, loss, the average AUC, and the temporal cost of the conducted experiments using various keyframe selection methods. We defined two baseline experiments: One is training the CNNs using all original frames of each video. The results of this experiment are shown as a baseline in Table 1. The second baseline is the training CNN using all eigenframes of each video. The experimental results of this baseline are shown as a baseline in Table 1. Table 1. Comparing the FER classification performance of various frame selectionmethods with the proposed method, RML database Classifier No. of frames Selection method Acc, % Loss AUC Time, s Offline Online 20 eigenframes 97.22 0.3837 0.9978 14186 9 20 k-means 91.89 0.3609 0.9387 17016 10 VGG-16 20 random 89.33 0.4819 0.9067 13205 8 baseline all frames 95.83 0.3272 0.9695 33917 107 baseline all eigenframes 91.67 0.3403 0.9499 38888 114 20 eigenframes 93.06 0.3847 0.9731 10679 5 20 k-mea