Geometric positions and optical flow based emotion detection using MLP and reduced dimensions

IET Image ProcessingVolume 13, Issue 4 p. 634-643 Research ArticleFree Access Geometric positions and optical flow based emotion detection using MLP and reduced dimensions Gulraiz Khan, Corresponding Author Gulraiz Khan gulraiz.khan@kics.edu.pk Al-Khawarizmi Institute of Computer Science, UET Lahore, PakistanSearch for more papers by this authorAiman Siddiqi, Aiman Siddiqi Al-Khawarizmi Institute of Computer Science, UET Lahore, PakistanSearch for more papers by this authorMuhammad Usman Ghani Khan, Muhammad Usman Ghani Khan Al-Khawarizmi Institute of Computer Science, UET Lahore, PakistanSearch for more papers by this authorSamyan Qayyum Wahla, Samyan Qayyum Wahla Al-Khawarizmi Institute of Computer Science, UET Lahore, PakistanSearch for more papers by this authorSahar Samyan, Sahar Samyan Al-Khawarizmi Institute of Computer Science, UET Lahore, PakistanSearch for more papers by this author Gulraiz Khan, Corresponding Author Gulraiz Khan gulraiz.khan@kics.edu.pk Al-Khawarizmi Institute of Computer Science, UET Lahore, PakistanSearch for more papers by this authorAiman Siddiqi, Aiman Siddiqi Al-Khawarizmi Institute of Computer Science, UET Lahore, PakistanSearch for more papers by this authorMuhammad Usman Ghani Khan, Muhammad Usman Ghani Khan Al-Khawarizmi Institute of Computer Science, UET Lahore, PakistanSearch for more papers by this authorSamyan Qayyum Wahla, Samyan Qayyum Wahla Al-Khawarizmi Institute of Computer Science, UET Lahore, PakistanSearch for more papers by this authorSahar Samyan, Sahar Samyan Al-Khawarizmi Institute of Computer Science, UET Lahore, PakistanSearch for more papers by this author First published: 08 March 2019 https://doi.org/10.1049/iet-ipr.2018.5728Citations: 5AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Recent times have witnessed an exponential increase in multimedia specifically visual contents. Emotions are considered an essential part for extracting facial features, evaluating the expressions and as a result predicting the emotions of any person is a trending topic of the time. Based on still images and consecutive video frames, a methodology has been proposed to anticipate the emotions. Facial action coding system (FACS) standards are utilised in the development of an automated visual based emotion detection system worldwide. Employing FACS, the authors estimated facial muscle movement by computing 24 landmark points, 16 mutual distances between them and wrinkles caused due to changing expressions. Canny edge detection has been deployed to calculate the intensity of wrinkles. Geometric positions and optical flow are the key methods deployed in the implemented methodology. The methodology was evaluated on self-generated, JAFFE dataset and EmotioNet. 1 Introduction Video contents are the highest growing medium across all data sources these days. There is a dire need to build systems which can process and automate these visual contents. Humans and their emotions are quite an interesting area of research which enjoys a part when it comes to interpretation of video sequences. Successful recognition of human emotions in video sequences can lead to a plethora of useful applications such as automatic surveillance systems, in the area of entertainment, gaming, medical testing of psychic patients by medical personnel and for evaluating the situation handling capabilities of a person. Facial expression detection is also being used in other specialties like lie detection and security [1]. Voice notes, body gestures and facial features are the most common methods for detection of emotions. Among these mechanisms, facial based recognition is the most popular approach since it is the approach followed by humans themselves for emotion recognition. Detection of emotions using images and video frames dominates detection using voice notes in a way that most security checks are carried out using surveillance cameras which are expected to record video sequences within a more profound manner as compared to voice notes. Most rich channels for emotion detection is through facial expression in visual content. Ekman studied human emotions in detail and found emotions are mostly depicted by facial expressions [2, 3]. Facial expressions are useless for conveying information that can be easily transferred through words, but facial expressions largely specify emotion detection that is difficult using words [4]. Facial expressions are a rich descriptor of smile, sadness, surprise, anger, fear and disgust because all these classifications involve significant facial muscle movement [4]. Emotion detection using facial features can be done by several methods ranging from simple rule-based methods to complex machine learning algorithms. Both appearance and geometric position based approaches are used for emotion detection like [1] and some techniques use an amalgam of above two as in [2]. Both of the above-mentioned approaches have their advantages and disadvantages, and we expect that systems that use both will ultimately result in the highest accuracy. Different algorithms are used for appearance-based techniques like Gabor representation of the face is formed and then processed by a bank of support vector machine (SVM) classifiers [3], local binary pattern (LBP) features are used for recognising facial expressions. Recent studies have revealed that emotion detection from facial expressions [3] has been an active area of research which has attained interest across the globe. Facial expressions and gestures dominate voice and other means of recognising emotions [1] since this is the method which humans themselves apply for recognising other emotions. Facial action coding system (FACS) proposed by Ekman and Friesen [5, 6] has become a world-wide standard for recognition of facial expressions based on muscle movements. Each of the action units (AUs) in FACS corresponds to a specific facial muscle movement that is used both individually and in combinations to define a particular facial expression [7]. This paper presents a novel method for the automatic detection of human facial emotions in still images, video sequences and live streams. This is achieved using facial feature points, mutual distances between them and the deviation of these points from their mean position over consecutive frames. The main theme of the discussed implementation is an analysis of geometric positions and optical flow algorithm and later, combining the results via probabilistic neural networks (PNNs) and bagging. The proposed system works effectively for live streams from webcams. The response of our system is real time with a frame rate of 25 per second and it processes each frame at runtime. Observation shows that most work in the field of emotion detection is based on video sequences directly [8, 9] but our effort is to detect emotions in still images and furthermore extend the methodology to keep track of changing facial expressions in consecutive video frames using the knowledge already extracted from these still images. Evaluation of the proposed method has been performed in two flavours (i) state-of-the-art dataset in the domain of emotion recognition and (ii) self-generated dataset which provides scalability of the proposed methods. The methodology has been evaluated on a dataset generated by CVML lab, KICS, University of Engineering and Technology, Lahore and a freely available expression database, i.e. Japanese female facial expression database [10]. The remaining of this paper is segmented as follows. Section 2 comprises of the literature survey of the previous work. Section 3 describes FACS. Section 4 elaborates the methodology. Section 5 gives the training followed by evaluation in Section 6. Conclusion and future work for emotion detection constitutes the last section of this paper. 2 Literature survey There has been plenty of work in the facial recognition field with different approaches [10]. Ekman initially started research on emotion recognition based on facial features of human [4]. Emotion detection from facial features can be accomplished by following FACS [4]. FACS has become a worldwide standard for the recognition of emotions [11-13]. Shichuan in his work extracts 94 fiducial points to define the external boundary of the face along with the boundaries of all the facial parts [11]. The relative displacements of these points from their positions in the neutral face in both horizontal and vertical directions act as the source of features for classification into seven emotions. Valstar et al. [12] use an approach based on AUs detection and intensity estimation. Just as methodologies implemented by many other researchers across the globe, Valstar has also utilised the technique of extracting 49 fiducial points before processing any further. Separate sets of these points are then made for eyes, eyebrows, mouth and other regions, for each of which Euclidean distances and angles amongst these points are computed. Later, geometric features and local binary Gabor patterns are taken into account for final results of emotion detection. Lucey et al. [14] make use of active appearance models (AAMs) to keep track of the face region from the entire image. The features have been extracted based on similarity normalised shape and canonical normalised appearance. The features are then fed into the SVM for classification. Tagliasacchi [15] explains the use of Lucas Kanade optical flow algorithm for motion detection. His publication presents various optical flow algorithms. Psaltis's [16] work is based on determining facial landmarks and then Lucas Kanade optical flow algorithm is used to track the movement of these points in consecutive frames to detect the emotions based on the deviation of a pixel from its initial location. From early 90s, the values of these AUs alone and their combinations are being employed with great accuracy to enable a system to detect facial expressions. Pantic and Rothkrantz [17] and Fasel and Luettin [18] have reviewed available facial expression recognition algorithms. Cohen et al. [19] introduced emotion recognition from facial expressions using multilevel hidden Markov models. Another widely used approach to detect emotion is through voice signals. Anagnostopoulos et al. [20] provide a survey of speech based emotion recognition. PNNs, deployed in the system, classify by the combined information from the probability density function (PDF) and the results of the feed-forward neural network [21]. The use of PNNs has been seen incorporated in methodologies for lip detection and tracking and for the classification of AUs [22]. As an output, true is returned for instance with maximum probability and false for the remaining cases. Bootstrap aggregation (bagging), a voting algorithm [23], has also proved useful in the detection of facial expressions. Bagging involves selection of train data based on random withdrawal, irrespective of any order or track of repetition of data items extracted once. This means that there may be a recurrence of the same data or the data item may be ignored altogether in the train set [24, 25]. Later, depending upon the similarity of the test instance with various classes, votes are assigned. The greatest number of votes implies the final output. Emotions recognition in smart phones is a quite challenging task. Hossain and Muhammad [26] have proposed a system for detecting emotions from images captured using the embedded camera of smart phones. After extracting face region, the bandlet transform is applied to split the face into non-overlapping regions followed by local binary patterns' (LBP) histograms to predict depicted emotion. Their system works well for the constrained dataset and gives an accuracy of 99.8% for JAFFE dataset. They achieved high accuracy for specialised systems in which testing set and training set were from the same environment. On the contrary, our system attains nearly 93% accuracy for JAFFE dataset and 94.7% for the real-time generated dataset. The slight difference in the two accuracies for JAFFE is due to generalisation. The system proposed by us handles more generalised situations and avoid over-fitting. Many other detection systems use computer vision to predict specific class. Wang et al. [27] present the locality constraint distance metric learning approach for detecting levels of traffic congestion in the scene. To incorporate real traffic scene, they trained their system on a realistic dataset containing 20 different scenes. Their system solves the congestion problem as a regression problem with congestion levels ranging from 0 to 1. Mean squared error is employed for evaluation purpose and achieved 0.076 error rate. Before emotion detection face localisation is required. We have employed a face detection method proposed by Viola and Jones [28]. There are some new localisation methods like Wang et al. proposed detection of roads in the scene using state-of-the-art Siamesed fully-convolutional networks [29]. They used two stream networks to processes the original RGB image and semantic contour map, respectively. KITTI dataset [30] is employed for training network and gained 98.5% accuracy. A comparison of previous work done on emotion detection has been given in Table 1. Table 1. Comparison of previous work Algorithm used Preprocessing Features Post Processing Dataset Accuracy histograms of extended LBP for low-resolution facial expression resolution (Shan et al. [8]) saving template, resizing image LBP features LDA MMIJAFFEtrain: CKTest: MIMI 86.979.850 HOG to extract the appearance features by accumulating the gradient magnitudes (Dahmane and Meunier [31]) face detection, contrast amelioration, brightness normalisation, scaling to 200 by 200, eyes detection HOGs PCA GEMEP-FERA 70 AAM SVM (Lucey et al.[14]) images captured and digitalised into 640 × 490 or 640 × 480 similarity normalised shape features, canonical appearance features using AAMs loose comparison whether any necessary AU was missing CK+ 83.1 multi-scale Gaussian derivatives combined with SVMs (Jain and Crowley [32]) face detection and standardising to 64 × 64 half-octave Gaussian pyramid construction multi-scale Gaussian derivative PCA GENKI 4k, CK 92.7 geometric positions and optical flow (our method) face detection, standardising to 320 × 240 and face alignment fiducial points mutual distances and wrinkles kernel locally linear embedding self-generated, JAFFE 94.7 (self) 92.16 (JAFFE) 3 Facial action coding system Ekman and Friesen [4] introduced standards for facial muscle movement which found application in almost all the facial expression deduction systems across the research world. This work was focused on alpha-numeric codes known as action units (AUs). Fig. 1 illustrates some of the basic AUs and their elaboration [33]. Table 2 describes the six basic emotions regarding AUs and their combinations. Table 2. Combination of various AUs for basic emotions [7] Emotion AUs Muscle movement smile 6 + 12 cheek raised and lip corner pulled sadness 1 + 4 + 15 brow lowered with an inner eyebrow raised and lip corner depressed surprise 1 + 2 + (slight) 5 + 26 eye brows raised, lids raised and jaw drop with opened mouth fear 1 + 2 + 4 + 5 + 7 + 20 + 26 eye brows raised, lids raised, lids tightener, lip stretched and jaw dropped anger 4 + 5 + 7 + 23 brow lowered, upper lip raised and lip tightener disgust 9 + 15 + 16 nose wrinkled and lips depressed Fig. 1Open in figure viewerPowerPoint FACS AUs 4 Methodology This work makes use of fiducial points for the task of emotion recognition. Initially, faces are passed through the pre-processing stage to combat the effects of noise and improve the quality of the input image. Fiducial points are then calculated using well-defined work in the literature of FACS. The Euclidean distance between these points is calculated for classification purposes. Wrinkles are extracted and distance based methods are applied to find emotions using these features. Later on, optical flow methods are used to counter the effect of motion in the recognised facial emotions. Further, the dimensional reduction is applied to cater the best features for emotion detection. Finally, we present a hybrid approach which is a mixture of all the above-mentioned methods and achieves comparable results with the state-of-the-art methods in emotion recognition domain. 4.1 Preprocessing The presence of noise in captured image/video can lead to poor results. Noise should be minimised if not removed altogether from the available datasets. Proposed system works well for static images and dynamic videos. The input of the system can be a static image, video clip or live streaming. In the case of a video sequence, it is first converted to individual frames [https://www.ffmpeg.org/]. Viola and Jones method based on Haar cascade classifiers which attain an accuracy of around 94% is applied to crop faces from the input images [28]. Extracted faces neither contain uniformity of size and dimensions across various frames, nor they are aligned in consecutive frames. To counter this problem, the following steps are performed: Images are resized to a fixed dimension, i.e. . Resizing make it possible for images with any dimensions to have consistent geometric features. By computing the angle of rotation from the mean position for each eye, the face regions are aligned (1)Fig. 2 presents a sample image after alignment. As can be seen, the frontal face looks more straight and ready to move to the next stages of features extraction. Fig. 2Open in figure viewerPowerPoint Face alignment Once these steps are done, the consequent cropped and resized frames become eligible for further processing, be it for the distance calculation algorithm or the optical flow algorithm for dynamic emotion detection, and are passed to the next stages of the system. 4.2 Fiducial points retrieval The use of fiducial features in this work is twofold; (i) extraction of fiducial points based onthe guidelines specified by AUs and (ii) computing the mutual distances betweencertain facial landmarks. Emotions are predicted based on the categorisation ofFACS AUs [4]. The points of interest are demonstrated in Table 3 and corresponding Fig. 3. Table 3. Extracted facial points Name Number Description LEB1 1 outer point of left eyebrow LEB2 2 inner point of the left eyebrow REB1 3 inner point of the righteyebrow REB2 4 outer point of the righteyebrow LE1 5 upper outer point of the lefteye LE2 6 upper inner point of the lefteye LE3 7 lower outer point of the lefteye LE4 8 lower inner point of the lefteye RE1 9 upper inner point of the righteye RE2 10 upper outer point of the lefteye RE3 11 lower inner point of the lefteye RE4 12 lower outer point of the lefteye NS1 13 lower middle point of the nose LIP1 14 left mouth corner LIP2 15 left upper point of mouth LIP3 16 middle upper point of mouth LIP4 17 right upper point of mouth LIP5 18 right mouth corner LIP6 19 right lower point of mouth LIP7 20 middle lower point of mouth LIP8 21 left the lower point of mouth CPLE1 22 central point of the left eye CPRE1 23 central point of the right eye CPLIP1 24 central point of mouth Fig. 3Open in figure viewerPowerPoint Points of interest in the facial region For extracting facial landmarks, the system has used open source Dlib library [34]. It is a cross-platform library written in c++. Dlib utilises a histogram of oriented gradients (HOGs) and a linear SVM classifier to detect the position of 68 fiducial points [35]. The cropped face is passed to dlib to infer all 68 facial points distributed among the complete allocated face. Our methodology uses 24 landmark points out of which 21 points are from a collection of 68 points extracted from dlib and remaining three are computed separately. Using the fiducial points of mouth and eye regions, midpoints of mouth and eyes are computed by taking into account the lines resulting from the four corner points and their intersection using slope computation and standard line equations. The midpoint of eyes and mouth is calculated using a line intersection algorithm as shown in Fig. 4. These midpoints are further used to calculate the distance between eyes and mouth midpoints. The consequent equations (11) and (12) calculate the central point (, ) of the left eye. The same mathematics applies to the right eye and mouth for the computation of their respective mid points. Fig. 4Open in figure viewerPowerPoint Computation of mid-point Using a generic equation of the line (2) we can derive lines and . (2) (3) (4)To find the midpoint of the intersection of these two lines, we equate generalised equations (5) (6)By putting values of and in (3) and (4), we get two equations regarding y and x. (7) (8) (9) (10)By equating (9) and (10), we get and (11) (12)where and are the slopes of lines and defined by (13) and (14), respectively, (13) (14) 4.3 Distance based methodology Initially, 14 distances from to as shown in Table 4 were calculated using Euclidean's formula. These distances were inclusive of raising of eyebrows, expansion of eyes, the opening of the mouth, expansion of mouth and distances between central points of eyes and mouth. Table 4. Distances Distance number Description Distance d1 raising of an outer left eyebrow d2 raising of an inner left eyebrow d3 raising of an inner right eyebrow d4 raising of an outer right eyebrow d5 opening of the left eye based on outer points d6 opening of the left eye based on inner points d7 opening of the right eye based on inner points d8 opening of the right eye based on outer points d9 opening of mouth based on left points d10 opening of mouth based on middle points d11 opening of mouth based on right points d12 expanding of mouth d13 distance of central points of the left eye to mouth central point d14 distance of central points of the right eye to mouth central point d15 distance of the lower middle point of the nose to left mouth corner d16 distance of the lower middle point of the nose to right mouth corner The displacement between these feature points is measured for calculating geometric features to find facial muscle position. These muscle movements further define current emotion as described in Table 2. The displacement between two-pixel values is defined as the number of direct pixels between the first point and second point. As lips can be displaced in both the horizontal and vertical directions, distances for lips are decomposed in horizontal and vertical displacements. For the opening and expansion of lips, vertical and horizontal displacements are considered, respectively. Moreover, only the magnitude of distance is considered in the final feature set. Fig. 5 provides the complete demonstration of distances between points. Fig. 5Open in figure viewerPowerPoint Computed distances The distances are calculated by using the Euclidean distance formula for n space points as given by (15), where and are any two points in n space, n being the number of dimensions. However, since our system deals with a two-dimensional space, (15) reduces to (16) (15) (16) 4.4 Distance and wrinkle based methodology There are some emotions like anger and disgust that require appearance-based features along with geometric features to classify facial expressions accurately (Fig. 6). Moreover, in case of sad expression, lip corners are depressed (see Table 2, Fig. 6c). In previous methodology, lip depressor was not catered by any of the distances. To incorporate lip depressor for detecting sadness, following two distances have been made part of the feature set: The distance of central nose point to left point of mouth, . Fig. 6Open in figure viewerPowerPoint Sample images for (a) Anger, (b) Disgust, (c) Sad The distance of central nose point to right point of mouth, . So overall this methodology is an extension of the distance based methodology employing an increased number of features regarding wrinkles and mutual distances between facial landmarks. The need to include wrinkles was introduced by Friesen and Ekman [36], stating that wrinkled nose and the presence of wrinkles between two eyebrows are the signs of disgust and anger, respectively, as shown in Fig. 6. The wrinkles of the nose and the region between the eyebrows are detected using Canny edge detection. Canny edge detection is a multi-step approach to detect edges in a given image. It applies Gaussian filter for noise removal and image smoothing. Gaussian filter for kernel size is given below: (17)where and . The intensity of gradient magnitude and angle after applying the Gaussian filter are computed by using the following formulae: (18) (19) 4.5 Optical flow-based methodology Optical flow keeps track of pixel movement over a series of frames, pointing out the direction of motion of pixels over time. This idea proves to be highly beneficial in monitoring the coordinates of a point over successive frames. Manipulating this idea yields effective results in the domain of dynamic emotion detection from video sequences by following the displacement of fiducial features in contiguous video frames. To account for the movement of the fiducial points, it is required to keep track of these points over consecutive frames, for which we make use of the optical flow algorithm. Each of the 24 points is tracked in 8 adjacent frames to even out the change in the coordinates of these points. We took eight frames because of frames more than eight results in discarding significant changes in the video sequence. On the contrary, a selection of a smaller count than eight leads to unnecessary computation for image frames that have negligible movements. Equations (20) and (21) show how this change is accounted for (20) (21)where n = 24 fiducial points. , represent the fiducial points. Optical flow acts as a fine tuning step of features. Final positions of fiducial points are determined using (20), where each new fiducial point is determined by adding its original value in the initial frame of a group and change tracked by optical flow in consecutive frames of a group. It is also utilised to incorporate the intensity of emotion. 4.6 Dimensions reduction There may occur redundancy in features causing errors while classifying and later predicting the correct instance. This high-dimensional data costs both the speed and accuracy of the classification algorithm. There is a need to convert these high-dimensional data into a lower-dimensional space to achieve better speed and accuracy [37]. Dimensionality reduction (DR) accounts for efficient data analysis by reducing the feature space compactly before passing it to the classifier. To detect emotions in images efficiently by minimising noise and ignoring corner cases, DR is one of the prime and essential actions for complex multi-dimensional input data. The goal of DR is to provide a more concise form of actual data. It is an illustration that captures all the information necessary for high-level decision-making. In DR, multiple DR techniques are tested on our extracted features using an open source library named Tapkee [38]. Main reasons for dimension reduction of our features are Noise reduction. Data compaction. Efficient feature extraction. Map data to lower-dimensionality, for data distribution recognition. There are many techniques for reducing the dimensions of data. Convolutional 2D linear discriminant analysis (LDA) is used in [39] for dimension reduction. Classification accuracy of 99.7% is achieved using this method. Considering convolution as resource hungry processes, we have used a novel technique that is gaining popularity, known as kernel locally linear embedding (KLLE) that apply dimensionality-reduction to the data for learning and classification. The need to discover compact representation of features allows the system to train itself on relatively less and more precise features. Here we used locally linear embedding, an unsupervised learning technique that computes low dimensions of data by preserving local relation of each data item with neighbours. LLE maps input vector to a single global coordinate system with low dimensions. LLE learns the global form of the non-linear manifold by using local symmetries of linear reconstruction. LLE can cast as kernel principal component analysis (PCA). The connection between LLE and kernel PCA is defined in [40]. Let be the greatest eigenvalue then KLLE can be defined in (22) (22)where (23)and I represents the input image. Features extracted from our emotion detection system are given as input to KLLE and it preserves the local linear relations in the input data which is encapsulated in a weight matrix W. The algorithm starts with constructing a neighbourhood graph by n nearest neighbouring and then W is obtained by the following equation: (24) is the vector of the reconstruction weights of , i.e. and is the local corre