Facial expression recognition using singular values and wavelet‐based LGC‐HD operator

IET BiometricsVolume 10, Issue 2 p. 207-218 ORIGINAL RESEARCH PAPEROpen Access Facial expression recognition using singular values and wavelet-based LGC-HD operator Durga G. Rao Kola, Corresponding Author Durga G. Rao Kola ganga_408@yahoo.co.in Department of ECE, UCEK, JNTUK, Kakinada, India Correspondence Durga Ganga Rao Kola, Department of ECE, UCEK, JNTUK, Kakinada 533003, India. Email: ganga_408@yahoo.co.inSearch for more papers by this authorSrinivas K. Samayamantula, Srinivas K. Samayamantula Department of ECE, UCEK, JNTUK, Kakinada, IndiaSearch for more papers by this author Durga G. Rao Kola, Corresponding Author Durga G. Rao Kola ganga_408@yahoo.co.in Department of ECE, UCEK, JNTUK, Kakinada, India Correspondence Durga Ganga Rao Kola, Department of ECE, UCEK, JNTUK, Kakinada 533003, India. Email: ganga_408@yahoo.co.inSearch for more papers by this authorSrinivas K. Samayamantula, Srinivas K. Samayamantula Department of ECE, UCEK, JNTUK, Kakinada, IndiaSearch for more papers by this author First published: 14 February 2021 https://doi.org/10.1049/bme2.12012Citations: 6AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Facial expression recognition (FER) is a significant research area in the human–computer interaction. The performance of FER systems depends on an efficient feature extraction method. This research work proposes a robust method for feature extraction using local gradient coding based on horizontal and diagonal (LGC-HD) operator, wavelet transform and singular values. The proposed framework consists of three steps in feature extraction. First, the wavelet transform is applied on facial images and features are extracted by applying the LGC-HD operator on low–low bands at different levels of decomposition. Second, the singular values of the facial image are calculated by applying singular value decomposition and these are used as features. Third, the features from the previous two steps are concatenated to obtain the final feature for FER systems. The proposed method combines the two advantages: (1) the integration of wavelet transform and LGC-HD operator provides discriminating and robust features. (2) The singular values are not sensitive to grey scale changes caused by the noise. After obtaining the features, the support vector machine is used for expression recognition. Extensive experimentation on Japanese Female Facial Expression database, Cohn–Kanade, FEI face and facial expression research group databases achieves good recognition rates of 84.2%, 92.3%, 98.2% and 97.9%, respectively. 1 INTRODUCTION Facial expression recognition (FER) is an important topic in human–computer interaction. FER has potential applications in human behaviour understanding, animation and driver mood detection [1]. Facial expressions are helpful to understand the human mood immediately [2]. The basic model of the FER systems is shown in Figure 1. The FER systems mainly consist of three parts: (1) face detection from given image/video, (2) feature extraction and (3) classification. Most of the FER systems classify the features into seven basic facial expressions: neutral, fear, happy, sad, anger, disgust and surprise. However, all FER systems not necessarily classify all of the facial expressions. The classification of facial expressions depends on the application of FER systems. For example, in medical application, FER system tries to detect the pain of the patient. For such cases, the problem becomes two-class expression recognition. FIGURE 1Open in figure viewerPowerPoint Facial expression recognition system Pre-processing is a very important step in FER systems before feature extraction. It involves three stages: (1) the face is detected from a given image, (2) after detecting the face, the face area is cropped to obtain the face region which eliminates any background in the given image and (3) all the cropped regions are resized to a fixed size. In this work, all images in the databases are resized to 128 × 128 pixels. Feature extraction plays a crucial role in FER systems. There are two kinds of approaches for feature extraction from the given face image. The first approach is geometric-based methods and the next approach is appearance-based methods [3]. Geometric-based approach requires the accurate detection of location of eyes, nose and mouth regions. But the feature extraction using geometric-based methods in real time environment is difficult. Hence, appearance-based approach gained popularity in feature extraction for FER systems [4]. In this approach, the features are extracted by applying the filters like Gabor filters for various spatial scales and orientations, local principal component analysis (PCA) etc., on some regions of face or whole face. Some popular appearance-based methods are PCA, independent component analysis and Gabor wavelets [5]. Among these methods, Gabor wavelets were powerful for efficient feature extraction, but the computational complexity involved in feature extraction was high. Recently, local binary patterns (LBP) has been introduced for texture analysis, and also subsequently used for feature extraction in FER systems [6]. LBP-based methods were computationally simple and efficient, but their performance was poor in noisy conditions [7]. Tan and Triggs [8] proposed the local ternary pattern (LTP) to improve the noise robustness of LBP, but the selection of threshold for binary encoding was difficult. Jabid et al. [9] proposed the local directional pattern (LDP) which was a superior method for texture description in a facial image. In LDP, an eight bit binary code was assigned to each pixel by comparing the relative edge responses using kirsch masks in eight different directions. The drawback of LDP was that its performance depends on selection of prominent edge responses. Tong et al. [10] proposed local gradient coding based on horizontal and diagonal (LGC-HD) principle operator for an efficient feature extraction. The binary encoding was based on the horizontal and diagonal gradient prior principle. This operator not only reduces the computational complexity, feature dimension but also effectively capture the texture characteristics. But, its performance was also effected by noise as in case of LBP. Holder and Tapamo [11] proposed improved gradient local ternary pattern (IGLTP) which was an improved operator over the LTP. In this operator, spatial filtering was performed on image using 3 × 3 Scharr kernels. After that, the LTP was applied on filtered image to capture the texture information. Since, it used the LTP again; the performance was influenced by the selection of threshold as in LTP. Histogram of oriented gradients (HOG) was another popular descriptor for texture in object recognition. Eng et al. [12] extended the HOG for feature extraction in FER systems. Nigam et al. [13] integrated both HOG and wavelet transform to enhance the performance of FER systems. The major drawback of all these methods is that their performance reduces for noisy conditions. In recent research works, deep convolutional neural networks (DCNN) was considered as breakthrough in computer vision and applied to many practical problems. The main success of DCNN was that it integrates both learning and classification. Shan et al. [14] developed DCNN which was capable to discover the deeper feature description for automatic emotion recognition. Jung et al. [15] proposed two deep networks: deep neural network (DNN) and convolutional neural network (CNN) and experimentally verified that CNN performs better compared to DNN. Liliana [16] extended DCNN for FER task by detecting the occurrence of facial action units to improve the performance of FER systems. Aneja et al. [17] proposed DeepExpr which was a novel expression transfer learning technique from humans to animated characters. Minaee and Abdolrashidi [18] proposed deep learning approach based on attentional convolutional network and analysed the crucial regions for detecting the facial expressions. Jaiswal and Nandi [19] developed real-time emotion detection system by using CNN architecture. Ding et al. [20] developed FaceNet2ExpNet (FN2EN) which incorporates face domain knowledge to regularise the training of an expression recognition network. Network architecture, consisting of convolutional blocks followed by fully connected layers was adopted for expression net. Li et al. [21] proposed deep locality-preserving CNN (DLP-CNN) to improve the discriminative power of features for emotion recognition. Further, real-world affective faces database was introduced which contains about 30,000 facial images. Despite of several advantages of deep learning, there are some limitations: they require large number of samples for training and huge computational complexity. In this research work, a robust method for feature extraction is proposed by using LGC-HD operator, wavelet transform and singular values. The feature extraction is obtained by concatenating the features using LGC-HD operator on low-low (LL) bands of wavelet transform and singular values. This approach combines the two advantages: (1) The LGC-HD operator on LL bands makes the feature description as more robust. (2) The higher singular values are not sensitive to grey scale changes caused by the noise. The remaing part of this article is structured as follows: brief review on LGC-HD operator, discrete wavelet transform (DWT) and singular value decomposition (SVD) is discussed in Section 2. The proposed method is explained in Section 3. Experimental procedure is presented in Section 4. Discussion on simulation results is presented in Section 5. Conclusions are given in Section 6. 2 RELATED WORK 2.1 LGC-HD operator The LGC-HD operator was proposed by Tong et al. [10]. It was a powerful discriminative operator to detect the texture changes in an image. The main advantage of this operator was that it has less computational complexity. This operator uses local gradient coding, in which the binary encoding was given to horizontal, vertical and diagonal gradients. The LGC-HD operator for a given 3 × 3 mask shown in Figure 2 is defined as: LGC − HD = s ( g 1 − g 3 ) 2 4 + s ( g 4 − g 5 ) 2 3 + s ( g 6 − g 8 ) 2 2 + s ( g 1 − g 8 ) 2 1 + s ( g 3 − g 6 ) 2 0 (1)where s ( x ) = { 1 , i f x ≥ 0 0 , i f x < 0 (2) FIGURE 2Open in figure viewerPowerPoint 3 × 3 mask for local gradient coding based on horizontal and diagonal operator The procedure for computation of LGC-HD code for a given 3 × 3 mask is illustrated in Figure 3. The LGC-HD operator is applied on each and every pixel of a given image and LGC-HD codes are calculated. Then histogram of all LGC-HD codes is used as feature vector. FIGURE 3Open in figure viewerPowerPoint Local gradient coding based on horizontal and diagonal (LGC-HD) coding example 2.2 Singular value decomposition SVD was powerful tool in the area of image processing and it mainly used for dimensionality reduction, image restoration and face recognition [22]. For a matrix of size m × n, the SVD is defined as: A m × n = U m × m D m × n V n × n T (3)where, U and V are orthonormal matrices with sizes m × m and n × n, D is diagonal matrix consisting of singular values. The SVD can be extended to an image by treating it as a matrix and is decomposed as U, D and V. Then singular values are used as algebraic features in face recognition [23]. The singular values in diagonal matrix are in descending order and most of the energy contained in first few values only. The singular values have an important property that only lower values are influenced by the noise but the higher values are immune to noise [24]. This significant property makes the use of singular values as a robust feature for face recognition applications. 2.3 Discrete wavelet transform DWT was popularly used for time-frequency analysis in the area of signal and image processing [25]. It allows the multi-resolution analysis which was useful for the image processing applications like image denoising and image compression. The Haar wavelet is the mostly used wavelet transform due to its ease of computation and defined as [26]: Ψ ( t ) = { + 1 , 0 ≤ t < 1 2 − 1 , 1 2 ≤ t < 1 0 , otherwise (4) The Haar wavelet transform decomposes the image into four sub-images using horizontal and vertical filters namely low–low (LL), low–high (LH), high–low (HL) and high–high (HH) bands, respectively. The LL band contains approximate details whereas remaining bands contains detailed information. 3 PROPOSED METHOD The procedure for feature extraction using LGC-HD operator, wavelet transform and singular values is discussed in this section. 3.1 Feature extraction using wavelet-based LGC-HD operator The LGC-HD operator is computationally simple and effective texture descriptor for FER systems. This operator outperforms in noise-free conditions compared to the traditional methods like LBP and its variants. However, the performance of this operator is reduced in the presence of noise. The wavelet transform is a more useful tool for image processing in noisy environment. This is motivated to combine the advantage of LGC-HD operator and wavelet transform for robust feature extraction. The wavelet transform is successively applied on the image to decompose into different resolutions. Since most of the energy contained in LL band, the LGC-HD operator is applied on LL bands only. The Haar wavelet is considered in this work because it is conceptually simple and memory efficient. The main steps in feature extraction are described below: The Haar wavelet is applied on each facial image to decompose it into LL, LH, HL and HH bands. The LGC-HD operator is applied on the LL band of the decomposed image, and then histogram of LGC-HD codes is calculated. The LL band is further decomposed into higher levels and step (ii) is repeated on each level of decomposition. The histograms from all levels of decomposition are concatenated to obtain the final feature and it is represented as f1. For feature extraction using wavelet-based LGC-HD operator, all facial images are resized to 128 × 128 dimension. Then, facial images are decomposed upto four levels of resolution using Haar wavelet transform. This decomposition produces images of size 64 × 64, 32 × 32, 16 × 16 and 8 × 8, respectively. The feature extracted from one LL band using LGC-HD operator has feature dimension of 32. So, the total length of the feature extracted from LL bands of decomposed facial image has a dimension of 128 (32 * 4 = 128). The procedure for feature extraction using wavelet-based LGC-HD operator is illustrated in Figure 4. FIGURE 4Open in figure viewerPowerPoint Feature extraction using wavelet-based local gradient coding based on horizontal and diagonal operator 3.2 Feature extraction using singular values The direct application of SVD on image is computationally difficult because of its large dimension. Hence, the image is divided into non-overlapping sub-blocks and SVD is applied on each sub-block in order to extract the singular values [27]. The singular values have the important property that first few values are sufficient to represent the image and any other variation like noise is reflected in the lower values. This property is used for noise robustness and hence, first few singular values are only considered. The main steps for feature extraction are described below: The image is divided into non-overlapping sub-blocks and SVD is applied on each block. The singular values from each block are concatenated and these are represented as f2. The final feature f using wavelet-based LGC-HD operator and singular values is obtained by using: f = f 1 σ 1 , f 2 σ 2 (5)where, σ 1 is SD of f1 and σ 2 is SD of f2. For feature extraction using singular values, each facial image of 128 × 128 dimension is subdivided into 4 × 4 blocks. This process gives sub-images of 32 × 32 dimension. The subdivision of facial image eases the computation of SVD and allows extracting the features at micro level. Then, SVD is applied on each block and singular values from each block are concatenated. The selection of number of singular values for FER is analysed in Section 5 and it is observed that first eight singular values achieves good recognition rate. Hence, the length of feature using singular values is 128 (8 * 16 = 128). The procedure for feature extraction using singular values is illustrated in Figure 5. FIGURE 5Open in figure viewerPowerPoint Feature extraction using singular values The overall structure of the proposed method is illustrated in Figure 6. The facial regions from the input images are cropped and resized to 128 × 128 dimension. The feature extracted using wavelet-based LGC-HD operator has dimension of 128. Similarly, the feature extracted using singular values has dimension of 128. These two features are concatenated which results a total feature dimension of 256. Further, the effect of selection in levels of decomposition and number of singular values on FER is explained in section 5. After feature extraction, the features are classified into seven basic facial expressions. FIGURE 6Open in figure viewerPowerPoint Overview of the proposed method The novelty of proposed method lies in the feature extraction in FER system. The feature extraction by using LGC-HD operator on LL bands reduces the influence of noise on texture description of facial images. Further, the wavelet transform allows the multi-resolution analysis. Similarly, the use of higher singular values by truncating the lower values results in denoising. Hence, the feature extraction using wavelet-based LGC-HD operator and singular values is more robust and achieves good performance in FER systems. 4 EXPERIMENTAL SETUP The databases and the experimental setup to evaluate the proposed method for FER systems are explained in this section. The Japanese Female Facial Expression database (JAFFE) [28] and Cohn–Kanade (CK) [29] databases are popularly used in the research area of FER systems. Hence, these databases are used in this work to evaluate the performance of the proposed method. The brief description about these two databases is presented here: The JAFFE database has 10 female subjects having a total of 213 images with a dimension of 256 × 256 pixel resolution. These 213 images include all basic seven facial expressions namely: neutral, anger, disgust, fear, happy, sad and surprise. Some of the sample images from this database are shown in Figure 7. FIGURE 7Open in figure viewerPowerPoint Sample images from Japanese Female Facial Expression database The CK database consists of 96 subjects with image sequences for all basic facial expressions. The image sequence starts from neutral expression and ends with posed facial expression. Each image has a dimension of 640 × 490 pixel resolution. The CK database is popularly used in FER systems because of large number in subjects. For the experimental setup, first image and last three images from each sequence are selected, and a total of 1290 images are considered. Some of the sample images from this database are shown in Figure 8. FIGURE 8Open in figure viewerPowerPoint Sample images from Cohn-Kanade database The facial region from each image in the database is cropped and then it is resized to 128 × 128 dimension. Then, the proposed method is applied on each image to extract the feature. After feature extraction, the features are classified into seven basic expressions using classifier. The support vector machine (SVM) classifier is considered in this work [30]. The SVM is a mostly used classifier in the area of machine learning [31]. The performance of the proposed method is evaluated by using 10-fold cross validation procedure. In 10-fold cross validation, each data set is divided into 10 groups approximately and 1 group is used for testing, and remaining 9 groups are used for training. This procedure is repeated for 10 times and average of all recognition rates is considered as a final recognition rate. The recognition rate gives the percentage of correctly classified facial expressions to the total number of facial expressions. The confusion matrix is calculated to know the efficiency of the proposed method on individual facial expressions. In a confusion matrix, the diagonal elements characterise the truly classified facial expressions and off-diagonal elements characterise the misclassified facial expressions. Two important measures from confusion matrix are calculated, that is, sensitivity and overall accuracy. Suppose C = [ C i j ] is the confusion matrix with i rows and j columns, then sensitivity of each class k and overall accuracy are calculated as [32]: Sensitivity = C k k ∑ j C k j (6) Overall accuracy = ∑ i C i i ∑ i ∑ j C i j (7) The overall accuracy is useful to assess the performance of the classifier in terms of recognition rate for all classes. The sensitivity is useful to assess the recognition rate of individual class. All simulation results are obtained using MATLAB software. The classification learner app in MATLAB is considered for SVM implementation. The quadratic kernel is selected for SVM because it gives the higher recognition rate compared to other kernels. 5 RESULTS AND DISCUSSION In this section, the selection of levels of decomposition and singular values for proposed method is analysed. Then proposed method is evaluated on JAFFE, CK, facial expression research group (FERG) and FEI face databases. 5.1 Experimentation using wavelet-based LGC-HD operator In the first setup of experiments, the selection of level of wavelet decomposition to apply the LGC-HD operator on LL bands is studied on JAFFE database using SVM classifier. The image is decomposed upto four levels and the feature is extracted using wavelet-based LGC-HD operator. The simulation results are shown in Figure 9 for different levels of decomposition. It is observed that the higher recognition rate is achieved at level 4 which is 83.2% for 6-class and 81.4 % for 7-class. The reason for high recognition rate at level 4 is due to the consideration of all LL sub-bands from highest to lowest resolution for feature extraction using LGC-HD operator. FIGURE 9Open in figure viewerPowerPoint Percentage recognition rate for different levels of decomposition 5.2 Experimentation using singular values In this section, the performance of the FER systems using singular values is studied. Each image of dimension 128 × 128 is divided into 4 × 4 blocks so that each sub-image has dimension of 32 × 32. The SVD on each sub-image is applied and singular values from each sub-image are concatenated. The SVM classifier is used to calculate the recognition rate. The recognition rate is calculated by considering first 8, 16, 24 and 32 singular values from each sub-image. The simulation results are shown in Figure 10 for various numbers of singular values. It is noticed that the higher recognition rates: 83.2 % for 6-class and 81.6 % for 7-class are achieved for the first eight singular values only. This is due to the fact that SVD contains most of the information in first few singular values only. Therefore, for the evaluation of the proposed method, the LGC-HD operator is applied on each LL sub-band upto fourth level and this is considered as a first feature vector. Similarly, the first eight singular values from each block are considered as a second feature vector. Then these two feature vectors are concatenated to get the final feature. FIGURE 10Open in figure viewerPowerPoint Percentage recognition rate for different singular values 5.3 Experimentation using different folds In this section, the proposed method is evaluated on JAFFE database for 5-, 10- and 15-fold cross-validation using SVM classifier. The simulation results for both 6-class and 7-class recognition are shown in Figure 11. It is observed that the highest recognition rates were obtained on the 10-fold cross-validation. The recognition rates are 84.2 % for 6-class and 82.6 % for 7-class recognition, respectively. Similarly, the same experimentation is performed on CK database for different folds of cross-validation. The simulation results are shown in Figure 12. It is noted that highest recognition rates were obtained on the 10-fold cross-validation. The recognition rates are 92.3 % for 6-class and 89.3 % for 7-class recognition, respectively. Since 10-fold cross-validation gives higher recognition accuracies, it is used in next experiments. FIGURE 11Open in figure viewerPowerPoint Percentage recognition rate for 5-, 10- and 15-fold cross-validation on Japanese Female Facial Expression database FIGURE 12Open in figure viewerPowerPoint Percentage recognition rate for 5-, 10- and 15-fold cross-validation on Cohn-Kanade (CK) database 5.4 Experimentation on JAFFE and CK databases In this section, the performance of the proposed method is verified on JAFFE and CK databases for 10-fold cross-validation using SVM classifier. The proposed method is compared with the traditional methods LBP [6], LTP [8], LGC-HD [10], IGLTP [11], HOG [12], Wavelet + HOG [13] and few deep learning methods: Deep CNN [14], CNN [15], DNN [15], Deep CNN [16], FN2EN [20] and DLP-CNN [21]. The results on JAFFE database are shown in Table 1 for both 6-class and 7-class expression recognition. The proposed method achieves a recognition rate of 84.2 % for 6-class and 82.6 % for 7-class expression recognition. From Table 1, it is concluded that the proposed method gives better recognition rate compared to other existing state-of-the-art methods. Similarly, the simulation results on CK database are shown in Table 2 for both 6-class and 7-class expression recognition. It is observed that the proposed method achieves a recognition rate of 92.3% for 6-class and 89.3% for 7-class. From Table 2, it is observed that performance of the proposed method is superior compared to LBP and its variants, HOG and some deep learning methods: Deep CNN [14], CNN [15] and DNN [15]. However, the performance of the proposed method is not superior compared to the deep learning methods: Deep CNN [16], FN2EN [20] and DLP-CNN [21] which achieves higher recognition rates of 92.8%, 98.6% and 95.8%, respectively. But these recognition rates are comparable with the proposed method in which the recognition rate is 92.3%. TABLE 1. Percentage recognition rate on JAFFE database Method 6-class 7-class LBP [6] 80.7 76.1 LTP [8] 82.9 79.5 LGC-HD [10] 83.2 81.3 IGLTP [11] 83.9 81.7 HOG [12] - 76.2 Wavelet + HOG [13] - 71.4 Deep CNN [14] - 76.7 Proposed 84.2 82.6 Abbreviations: CNN, convolutional neural network; HOG, histogram of oriented gradients; IGLTP, improved gradient local ternary pattern; JAFFE, Japanese Female Facial Expression; LBP, local binary patterns; LTP, local ternary patterns; LGC-HD, local gradient coding based on horizontal and diagonal. TABLE 2. Percentage recognition rate on CK database Method 6-class 7-class LBP [6] 88.3 85.6 LTP [8] 90.1 87.9 LGC-HD [10] 91.3 88.5 IGLTP [11] 91.7 88.9 Wavelet + HOG [13] 90.0 - Deep CNN [14] - 80.3 CNN [15] - 86.5 DNN [15] - 72.7 Deep CNN [16] - 92.8 FN2EN [20] 98.6 - DLP-CNN [21] 95.8 - Proposed 92.3 89.3 Abbreviations: CK, Cohn-Kanade; CNN, convolutional neural network; DLP-CNN, deep locality-preserving CNN; DNN, deep neural network; FN2EN, FaceNet2ExpNet; HOG, histogram of oriented gradients; IGLTP, improved gradient local ternary pattern; JAFFE, Japanese Female Facial Expression; LBP, local binary patterns; LTP, local ternary patterns; LGC-HD, local gradient coding based on horizontal and diagonal. The main advantage of proposed method compared to deep learning methods is that it requires less time to complete the task of FER systems. The complexity of the proposed FER systems is less compared to the deep learning-based FER systems. The confusion matrices are calculated to understand the performance of the proposed method on each individual facial expressions using SVM classifier. The simulation results on JAFFE database are shown in Tables 3 and 4 for both 6-class and 7-class express recognition. For instance, it is observed from the second row of Table 3 that disgust images are classified correctly with 96.6% recognition rate and remaining are misclassified as anger with 3.4% recognition rate. In 7-class testing, the performance is lower because of confusion with the neutral expression. Similarly, Tables 5 and 6 show the confusion matrices for CK database for 6-class and 7-class expression recognition. It is noticed that the higher recognition rates were obtained on CK database compared to the JAFFE database. The reason for these higher recognition rates is due to availably of large number of subjects in CK database. It is also observed that 7-class expression recognition is lower compared to the 6-class