Influence of shape and texture features on facial expression recognition

IET Image ProcessingVolume 13, Issue 8 p. 1349-1363 Research ArticleFree Access Influence of shape and texture features on facial expression recognition Asit Barman, Corresponding Author Asit Barman mtechitasit@gmail.com Department of Computer Science & Engineering and Information Technology, Siliguri Institute of Technology, Darjeeling, 734009 IndiaSearch for more papers by this authorParamartha Dutta, Paramartha Dutta Department of Computer & System Sciences, Visva-Bharati University, Santiniketan, 731235 IndiaSearch for more papers by this author Asit Barman, Corresponding Author Asit Barman mtechitasit@gmail.com Department of Computer Science & Engineering and Information Technology, Siliguri Institute of Technology, Darjeeling, 734009 IndiaSearch for more papers by this authorParamartha Dutta, Paramartha Dutta Department of Computer & System Sciences, Visva-Bharati University, Santiniketan, 731235 IndiaSearch for more papers by this author First published: 29 May 2019 https://doi.org/10.1049/iet-ipr.2018.5481Citations: 7AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Shape and texture features provide a novel framework for expression recognition by using salient landmarks based triangle initialisation and identification of texture regions. Detection of effective landmarks is achieved by well-known appearance-based model to form a grid. Accordingly, several triangles are identified within the grid with respect to the nose landmark reference point. These salient landmarks are also used to find the texture regions. Normalised shape and texture signatures are derived from triangles and texture regions. Stability indices are determined from shape and texture signatures which are also exploited as significant features for recognition of facial expressions. Statistical measures such as range, moment, skewness, kurtosis, and entropy are used to supplement the feature set. Individual and combined features are fed into multilayer perceptron and deep belief network (DBN) network to classify different expression categories encompassing anger, sadness, fear, disgust, surprise, and happiness. The authors investigated the performance of their proposed system on Cohn–Kanade, JAFFE, MMI, MUG, and Wild face benchmark databases. Thorough experimentation establishes the performance superiority of the proposed methodology over other existing competitors. Combining features is also validated through extensive results and analyses. 1 Introduction Facial expression analysis is a challenging research area in the field of affective computing. Affective computing [1] refers to a system capable of reading human emotions. Human emotions are reflected in the human face to a large extent during verbal and nonverbal communication. Facial expression recognition system has many applications encompassing the area of educational tutoring system, surveillance, video conferencing, and driving safety etc. In the health care system, a doctor can go for emergency treatment of a seriously ill patient upon his/her emotional state. The study of facial expressions began with the effort rendered by Ekman and Frisen [2]. They identified six basic facial expressions namely: fear, surprise, disgust, anger, sadness, and happiness. They took recourse to clues based on geometric features extracted from eyes, eyebrows, mouth, jaw etc. These clues were encoded into the Facial Action Coding System (FACS). FACS was found to produce a temporal profile of each facial movement in terms of action units (AUs). In this article, we introduce features such as shape signature, texture signature, and stability indices in addition to frequently used statistical features namely: skewness, kurtosis etc. These features together are fed to multilayer perceptron (MLP) and DBN for the recognition in the supervised mode of different facial expressions. Human face components such as eyes, eyebrows, mouth, jaw etc. play an important role in distinguishing one expression from another. Appearance-based method is applied to the human face images to identify the landmark points. Salient landmark points are considered in and around the regions of eyes, eyebrows, nose, and mouth. We consider three points on the eyebrow, four points on each eye, three points on the nose, and four points on mouth region. These salient landmarks constitute a grid and we identify different triangles within the grid. Shape and texture signatures are obtained from the grid and normalised. Subsequently, the stability indices are determined from the normalised shape signature and texture signature pair and this is found to play a vital role in recognising different expressions reflected on a face. Statistical measures such as range, moment, skewness, kurtosis, and entropy are calculated from the normalised shape and texture signature duo. The combination of all these features together is fed as input to the MLP as well as DBN for classification of expressions. We observe that the combined shape–texture (S-T) signature-based facial expression recognition system proposed here offers more discriminating results of classification of human expressions than individual shape and texture signature-based facial expression recognition. The motivations of this paper: our proposed procedure is different from the method in [3] where the authors used geometrical displacement of certain marked Candide node as the difference of the node between first (neutral) and greatest intensity facial expression and facial AUs as input of support vector machine on image sequences to classify the emotions. If there are many neutral expressions of the same person, different types of geometrical displacement may be produced for the same expression. It would create problems in the task of classification of expressions. The authors experimented the method only on Cohn–Kanade (CK) + [4] database. To overcome all these we consider the grid on face image and determine the triangles relative to the nose landmark point at the central position to form the shape signature. These salient landmark points are also considered for finding the texture signature. These shape and texture signatures individually and in the combined form are used as input of different neural networks such as MLP and DBN to classify the expression. The contributions to this paper consist of: (i) grid formation using salient landmark points; salient landmarks are considered on the eyes, nose, mouth, and eyebrows on a face image. These salient landmarks form a grid by connecting each other. (ii) Triangles are identified within the grid. These triangles are used to find the shape signature. (iii) Texture region formation: salient landmarks are extracted from eyes, nose, mouth, and eyebrows regions to form a texture region. These regions play a significant role to differentiate the expressions from one face to another face. (iv) Shape and texture signature: shape signature is computed from triangles and texture signature is computed from texture region. These signatures are used to recognise the expressions individually and in a combination of S-T. (v) Stability index: stability indices are calculated from shape and texture signatures. (vi) A number of features are expected to provide prominent facial expressions recognition. For this reason, relevant statistical measures such as range, moment, skewness, kurtosis, and entropy are considered to get prominent feature set. The remaining article is organised as follows. Section 2 presents an overview of relevant work. In Section 3, we introduce and elaborate on the features used for facial expression. Grid formation and triangle initialisation are discussed in Section 4. The training procedure is presented in Section 5. Experiment and result discussions are offered in Section 6. Finally, conclusions are drawn in Section 7. 2 Related work Proper landmark detection and extraction of salient features on facial images thereof are still challenging research propositions in facial expression recognition. Facial feature extraction mainly comprises geometric and appearance-based approaches. Geometric approaches are concerned with the geometrical position of eyes, eyebrows, mouth etc. These approaches are focused while finding out the displacements of facial points with respect to neutrality. In [5], the authors used an active shape model to locate the features like shape and texture information. These features were used to classify the expressions. The authors [6] proposed a new method of matching statistical models of appearance into images. A set of model parameters were used to control modes of shape and grey-level variation learned from a training set. They constructed an efficient iterative matching algorithm by learning the relationship between perturbations in the model parameters and the induced image errors. In [7], the authors proposed a very simple framework for deriving the well-known optimisation problems in active appearance models (AAMs) and most importantly for deriving efficient solutions. They pointed out some salient landmarks on the face images and extracted the prominent features to categorise the expressions on the basis of those landmarks. Boughrara et al. [8] presented a constructive training algorithm for MLP which was applied to facial expression recognition applications. The developed algorithm was composed of a single-hidden layer using a given number of neurons and a small number of training patterns. In [9], two-dimensional DCT was used to extract the features over a face image and a constructive one-hidden-layer feed forward neural network was put to use for recognising the expressions. In [10], the authors proposed a simple solution for facial expression recognition that used a combination of standard methods, like convolutional network and specific image pre-processing steps on a given feature set. Majumder et al. [11] proposed a deep network consisting of autoencoders and the Self Organizing Map (SOM)-based classifier for facial expression recognition. The fusion of geometric features with local binary pattern (LBP) features using autoencoders provides a better representation of the facial expression. The performance of the proposed approach acquired average recognition rate on two widely used databases to the tune of 97.55% for MMI and 98.95% for CK+. In [12], Gauss–Laguerre wavelets extracted the high-frequency texture information of various facial expressions. Authors also considered geometric positions of fiducial points offering valuable information for upper/lower face AUs. The combinations of these features were used for facial expression recognition. In [13], a two-stage multi-task sparse learning (MTSL) framework was reported to efficiently locate the discriminative patches to be used by a classifier to categorise the expressions. Happy and Routray [14] extracted discriminative features from salient patches of various expressions and a classifier was used to categorise the expressions into different categories. The authors [15] used a local directional position pattern (LDPP) to extract robust feature. The LDPP features are extended by principal component analysis and generalised discriminant analysis to make them more robust for better face feature representation expression. The proposed features are finally applied to the deep belief network (DBN) for expression training and recognition. In [16], the authors extracted geometric features from point, line, and triangle composed of tracking results of facial points. The most discriminative line and triangle features are extracted using feature selective multiclass AdaBoost with the help of extreme learning machine (ELM) classification. An orthogonal matrix and a sparse matrix [17] were considered to guarantee that the extracted features could act as the main features of the original data enhancing robustness to noise and thus robust sparse linear discriminant analysis is found to have the potential to perform better than other discriminant methods. Authors in [18] proposed inter-class sparsity-based discriminative least square regression for multi-class classification. 3 Proposed schematic presentation The proposed methodology of shape and texture signature-based facial expression recognition is shown in Fig. 1. Well-known AAM [6, 7] is applied on face images to determine the landmarks. Salient landmarks are considered on the eyes, eyebrows, nose, and mouth regions to extract prominent features. A grid is formed on the face images with these landmarks. Corresponding texture regions are extracted with the help of salient landmarks. As such because of its central geometric location on a human face, the landmarks near the nose region are considered as reference points to form triangles with other landmarks in any facial expression. The triangles are determined accordingly within the grid and any such triangle formed has a specific shape signature as per (3). It may be noted that the shape signature of a specific triangle is sensitive to the facial expression under consideration. We shall try to exploit this particular property to categorise the expression being considered. Shape signature information of different such triangles formed in a typical facial grid, considered in a particular sequence induces a shape signature. Corresponding texture regions are considered to calculate the texture signature using LBP. Normalised shape and texture signatures are also used to find respective stability indices which are found to play a vital role to differentiate the expressions. In addition, statistical features such as moment, range, skewness, kurtosis, and entropy are computed from normalised shape and texture signature pair to ensure prominence of enhanced feature set in respect of face images. Combinations of extracted features are used as input to MLP and DBN networks to categorise the expressions of shape and texture signature pair. Fig. 1Open in figure viewerPowerPoint Flow diagram of proposed methodology 4 Grid formation and triangle identification Landmark detection on the face under variation of facial expressions has been extensively studied in [19, 20]. It is observed that few landmarks are common during changes of basic expressions and some are very sensitive in differentiating one expression from another with respect to the face of the same person. Proper landmark detection is a challenging task for the researchers engaged in this task. AAM [6, 7] is applied to the face images to decide the landmarks. According to [21], the landmark points play an important role due to the displacement and distortion of the facial components during dynamic changes of the expressions. We use Fast-Sic [7] AAM fitting to get the landmarks on the facial image. Salient landmarks are considered on the eyebrows, eyes, nose, and mouth region due to their variation of expressions, as we are aware of the positions of landmarks. The prominent landmarks in different facial expressions are shown in Figs. 2-4. These salient landmarks are positioned on the eyes (two corner points, one mid position of upper eyelid, and another one being in the mid position of lower eyelid), eyebrows (two corner points and another mid position of two corner points), nose (two corner and one in mid position), and mouth (two corner points of lips, one in mid position of upper lip and another one in the mid position of lower lip) which are shown in Fig. 2. These 21 points are connected to each other and accumulated in an array in a specific order called grid as depicted in Figs. 5 and 6. The first row of a grid, considered as the first landmark, is compared with the second and then the first landmark is compared with the third landmark which continues until the last landmark is compared with first which is shown in Fig. 7. As such because of its central geometric location in a human face, the nose region landmarks are considered as reference points to form triangles with other landmarks in any facial expression. The triangles are determined accordingly within the grid and any such triangle which is depicted in Figs. 5 and 6. Fig. 2Open in figure viewerPowerPoint Landmark detection on CK+ of anger expressions Fig. 3Open in figure viewerPowerPoint Landmark detection on CK+ of sad expressions Fig. 4Open in figure viewerPowerPoint Landmark detection on CK+ of surprise expressions Fig. 5Open in figure viewerPowerPoint Grid and triangle initialisation on CK+ of different expressions Fig. 6Open in figure viewerPowerPoint Grid and triangle initialisation on JAFFE of different expressions Fig. 7Open in figure viewerPowerPoint Grid initialization 4.1 Texture region formation Active patches are obtained from the face image depending upon the variation in facial patches. These active facial patches are considered at eye corner, nose corner, and mouth corner regions due to their elicitation of expressions. The size of the active facial patches is kept at . The patches are numbered among their active regions as depicted in Fig. 8. Active facial patches are dependent on salient landmarks. Active patches and are directly extracted from eyebrows and eye corners, respectively, as shown in Fig. 8. and are the nose corner points. and are the lip corner points. and are located just below the lip corner and is the mid position of and , , and are adjacent to nose corner points. Now we combine all active facial patches to obtain a texture region. Fig. 8Open in figure viewerPowerPoint Position of facial patches 4.2 Local binary pattern LBP [22] is widely used for robustness to changes in illumination. This operator derives a binary number by using the neighbouring pixel values with the candidate pixel value. The bit pattern is calculated as (1)where n is the total number of neighbourhood, is the pixel value in the neighbourhood of and is the pixel value at coordinates and the LBP histogram of image LBP can be calculated as (2)where different labels produced by LBP operator are represented by p. In this work, the uniform pattern is used instead of basic LBP. A uniformity measure of LBP computes the number of bit-wise transitions from 1 to 0 and from 0 to 1 for a circular bit pattern. The patterns 01000000 and 00011100 are examples of uniform as both contain 2 transitions, while 00011010 and 00101000 are non-uniform as they contain more than two transitions. The uniformity measure of the pattern is at most two. In [22] it has been established that >90% of the information is preserved in (8,1) neighbourhood texture image. According to [22], the total number of the uniform pattern of is 59. 4.3 Feature selection Facial expression recognition depends upon the appropriate extraction of the features of human facial expressions. Extraction of discriminative features from salient landmarks plays the cardinal role of facial expression recognition. We shall elaborate on the different features used in our proposed recognition framework. 4.4 Shape signature and texture signature A grid is formed on a human face using salient landmarks. The possible triangles are identified within the grid. The perimeter , where x, y, z are the edges of the triangle, , and the area is calculated as . They are computed to determine the shape as (3)where index i is used to specify a particular triangle in the face grid. Texture region formation is discussed in Section 4.1. We only consider the LBP to extract the texture feature from the texture region. Normalised shape and texture signatures are computed as (4)where q is the total number of shapes or textures, and M is the total number of images. Here, is the shape or texture signature and are the shapes of triangles or texture regions as applicable. 4.5 Stability index Stability index also plays a vital role in the present context. Variations of stability indices on face images are shown in Fig. 9. The normalised shape and texture signatures are used to get the higher order signature as (5)where the order shape and texture signatures are computed for each face image indexed by f. Different are the shapes of triangles or texture regions. In this experiment, we used order as . We find their order signature differences as (6) Fig. 9Open in figure viewerPowerPoint Stability indices of MMI database We set a threshold upon the analysis of the differences in higher order factors. This threshold follows the condition . Such an order r is treated as a stability index of shape or texture signature. 4.6 Statistical analysis To increase the accuracy of facial expression recognition, we also consider statistical measures such as range, moment, skewness, kurtosis, and entropy of both shape signature and texture signature. The range is calculated from the differences between maximum and minimum values of the normalised shape as well as texture signature. The fourth order moment pair [23] is extracted from both normalised shape and texture signature using (7)where is the normalised signature vector and is i raised to the power of k, for non-negative integer values of k; . Asymmetry in the data is measured by skewness [23]. Skewness and kurtosis [24] are calculated as (8) (9)where U and V are the skewness and kurtosis, respectively. Entropy [25] is calculated as (10)where contains the histogram count of both shape as well as texture signature in normalised sense and , . 5 Feature classification Extracted features of each facial image using shape and texture signatures are represented by a vector of nine elements each. After the combination of both S-T signature, every face image is identified by a vector comprising of 18 elements. These features are used as Min–Max process matrices by normalising the minimum and maximum values of each row feature vector and also as input to MLP [8] and DBN [26] to get output responses in terms of minimum errors [27]. Proposed training algorithm of facial expression recognition using shape, texture, and S-T signatures is shown in Fig. 10. Fig. 10Open in figure viewerPowerPoint Proposed training algorithm for facial expression recognition using shape and texture signatures 5.1 Training using MLP MLP training is performed by reducing the mean square error [27] for multiple classes. Here and are the target and desired response and we set in the MLP. The training algorithm involves four steps: initialisation of weights, feed forward, back propagation of errors, and updating of weights and biases. During the first step, we assign small random values to initial weights. Input unit , where n is the number of the input received signal and transmits this signal to all hidden units . We also consider biases and . Each hidden unit computes the tan h activation function and dispatches its signal to each output unit. The output unit computes the output using a linear activation function. During backpropagation of errors, each output unit measures its calculated activation with its target value to calculate the associated error for that pattern with that unit. Depending on these errors, the factor is determined at the output unit and sent back to all the previous layer units. In this experiment m is the total number of output (here it is set to six). Similarly, the factor is calculated for each hidden unit , here is the number of hidden units. In the last step weights and biases are updated using factor and the activation. We also consider and as the threshold and the learning rate, respectively. The training is stopped when . During the testing, six basic expressions are assigned to the maximal value of the neuron outputs. The performance of the recognition system is measured in terms of recognition rate on the testing set. MLP tests the performance of recognition of facial expression falling under different categories. 5.2 Deep belief network training DBN [26] consists of two main parts: (i) first one is pretraining and (ii) the second one is fine-tuning. The pretraining phase follows the mechanism of Restricted Boltzmann Machine (RBM) [28]. RBM is very useful for unsupervised learning that reduces the local optimum error. The main characteristic of DBN is its proficiency to extract and select salient features from the input feature. Every layer of RBM is adjusted by being supported by the previous layer. When the first layer computes the weight matrices, these are considered as an input to the second layer and such mechanism continues in the same way until the last layer. This process continues to train RBM one after another. During the training process, the input is reduced layer by layer and hence, the selected features at the hidden nodes of the last layer can be considered as a vector of features for the current layer. The Contrastive Divergence [29] can be used to update the weight matrix layer by layer. DBN consists of three hidden layers which have a different number of neurons in different layers such as 9/18 for input layer, 8/16 for the first hidden layer, 6/12 for the second hidden layer, 4/8 for the third hidden layer, and six for the output layers showing to train and categorise six basic expressions. For initialisation of the network, a greedy layer-wise training methodology is applied. Once the weights of the first RBM are trained, the first hidden layer () becomes fixed. Then, the weights of the second RBM are modified for training using the fixed first hidden layer. Then, the third RBM is trained with the help of previous RBM. The process of training a typical RBM involves some crucial steps. First of all, initialisation is done where a bias vector for the visible layer P, a bias vector for the hidden layer H, a weight matrix T are set to zero. (11) (12) (13)Next, utilising (11), the binary state of the hidden layer is calculated and later on it is used to reconstruct the binary state of the visible layer using (12). Then, the hidden layer is re-computed given where (14)The threshold value is learned with the weights to determine the output of the sigmoid function in the network and the weight difference is computed as (15)where B is considered as batch size. Finally, the current weight becomes a summation of the previous weights. These steps are repeated for all the batches. When the RBM process is completed, a typical backpropagation algorithm is applied for adjustments of all parameters for fine-tuning. 6 Experiments and results We evaluate shape signature and texture signature-based facial expression recognition separately and in combination by applying on CK+ [4], JAFFE [30], MMI [31], MUG [32], and Wild [33, 34] databases. Extracted feature of each facial image using shape and texture signature is represented by a vector of 9 elements. MLP and deep neural network-based DBN have been used to categorise the expressions. We also compare the results with the different state-of-art methods in order to justify the effectiveness of the proposed method. The six basic expressions are denoted as anger (A), disgust (D), fear (F), happiness (H), sadness (S), and surprise (G). 6.1 Experiment on CK+ database The CK+ database consists of 100 university students aged between 18 and 30 years. This database has 65% female, 15% African-American, and 3% Asian or Latino faces. Extracted features are fed to MLP and DBN to evaluate the performances of facial expression recognition. To evaluate the recognition rate, the database has been considered into a training set and a testing set. A total of 1175 images are divided into 782 for training and 393 for testing. 6.1.1 Results evaluation of shape, texture, and S-T signature on CK+ database The proposed shape, texture, and S-T signatures use total 393 (out of them there are 71 anger, 44 fear, 54 disgust, 84 happiness, 71 sadness, and 69 surprise instances) expressions to test the performance of facial expression recognition. Table 1 shows the performance of different expressions of shape, texture, and S-T signatures using MLP. Shape signature classifies 70 images properly as anger but two images are misclassified with disgust, disgust classifies correctly 50 images but four images are misclassified with fear, fear classifies 41 images correctly and two images confuse with happiness, happiness classifies 82 images correctly and two images are confused with sadness, sadness classifies 69 images correctly and one image is misclassified with happiness, and surprise classifies 69 images correctly. Texture signature classifies correctly 68 images as anger, 47 images as disgust, 31 images as fear, 80 images as happiness, 67 images as sadness, and 69 images as surprise. In the same way, S-T signature shows the correct classification of expressions with some misclassification. In contrast, fear is the most difficult facial expression to be correctly recognised for texture signature. It is observed that shape and S-T signatures give promising expression recognition, whereas texture signature performs poor recognition rate of expressions for fear. In this experiment, common texture regions are shared with disgust when fear is classified. Table 1. Confusion matrix of shape, texture, and S-T signature on CK+ database using ML