# Facial Expression Recognition Using Kernel Entropy Component Analysis Network and DAGSVM

Authors: Xiangmin Chen; Li Ke; Qiang Du; Jinghui Li; Xiaodi Ding
Journal: Complexity
 (2021)

---

## Abstract

Facial expression recognition (FER) plays a significant part in artificial intelligence and computer vision. However, most of facial expression recognition methods have not obtained satisfactory results based on low-level features. The existed methods used in facial expression recognition encountered the major issues of linear inseparability, large computational burden, and data redundancy. To obtain satisfactory results, we propose an innovative deep learning (DL) model using the kernel entropy component analysis network (KECANet) and directed acyclic graph support vector machine (DAGSVM). We use the KECANet in the feature extraction stage. In the stage of output, binary hashing and blockwise histograms are adopted. We sent the final output features to the DAGSVM classifier for expression recognition. We test the performance of our proposed method on three databases of CK+, JAFFE, and CMU Multi-PIE. According to the experiment results, the proposed method can learn high-level features and provide more recognition information in the stage of training, obtaining a higher recognition rate.


---

## Body

## 1. Introduction

Facial expression recognition (FER) has great application potential in emotional computing, intelligent robot, intelligent monitoring, and clinical medicine [1, 2]. A FER system mainly consists of image acquisition, image preprocessing, image feature extraction, and classification [3], among which the image feature extraction directly exerts an impact on the training of classifiers and the performance of the whole recognition system. The effective feature extraction method can be used to improve recognition accuracy. In essence, expression feature extraction refers to the conversion of a high-dimensional facial expression image vector into a low-dimensional vector with a lot of distinguished information. Many algorithms used to extract image features are commonly applied to facial expression recognition [4–10].Among the numerous feature extraction algorithms, principal component analysis (PCA) [11] is essentially a classic K-L transform and widely applied to extract features from statistics in pattern recognition. Its principle is to find an optimal orthogonal transformation to minimize the mean squared error for transformed data. In general, PCA can represent the main information of the initial data by using a few principal components. However, PCA has weak robustness for some images with large illumination changes and complicated facial expression changes. In addition, PCA loses plenty of useful information because it cannot be used for nonlinear problem. As for the nonlinear problem case, the initial data can be mapped onto the higher-dimensional space through some nonlinear mapping. The linear dependence between the data in the high-dimensional space is utilized to fit the nonlinear dependence in the original data to the greatest extent, and then PCA can be applicable in this high-dimensional space. This is the kernel PCA or KPCA for short [12, 13]. But, it cannot learn high-level features and achieve good performance.Recently, deep learning (DL) [14] has achieved a great success in terms of computer vision and pattern recognition. The DL has been gradually applied to feature extraction of FER. The convolutional Neural Network (CNN) [15–18] is a very basic DL model and represents a sort of the artificial neural network. The structure of the CNN is similar to the connectivity model of neurons in the human brain, which makes the network model less complicated and minimizes weight amount. Such merit is more evident when the multidimensional image is inputted into the network. This model makes the image as an input into the network directly and reduces the complexity of feature extraction. However, the CNN has a complex hierarchical structure, slow convergence, and excessive dependence on the network’s initial [19–23].To fully utilize the structural features of the CNN and avoid its drawbacks, researchers integrate the feature extraction methods widely used in pattern recognition with the deep learning model, i.e, the deep subspace model. This model retains the deep learning idea, which can not only have powerful feature extraction ability but also reduce the computational burden. Liong et al. researched a deep learning model Deep PCA [24]. By using two-layer ZCA whitening (zero-phase component analysis whitening) and PCA networks, the new feature representation can be obtained by directly cascading the features learned at the first layer with those obtained at the second layer. The newly obtained features contain more discriminant information, which greatly improves the accuracy of image recognition. However, it shows sensitivity to a singular point, and the robustness of the algorithms for noise is poor. Chan et al. proposed PCANet structure [25], which cascades PCA, binary hash coding, and block histogram, to take the final output as the extracted deep feature. As an effective depth subspace model, the experimental results demonstrate that in terms of feature extraction, PCANet outperforms KPCA, PCA, and other methods and even AlexNet, ConvNet, and other CNN structures in some respects. Despite the demonstrated success, the model is restricted to the linear relation among the features and limits the use of features with nonlinear metrics.Motivated by these successes, based on the current subspace learning and feature extraction algorithms, we propose a novel facial expression recognition algorithm using KECANet and DAGSVM. The convolutional kernel group of the expression sample set is obtained by KECA, which prevents CNN’s excessive reliance on network initial value and requires a constant iterative solution. Then, the network output features are processed by binarization and block-histogram statistics to facilitate the extracted features to achieve better global discrimination. In addition, an effective classification method DAGSVM based on relative distance is applied to the classification.The rest of this paper is organized as follows: In Section2, an unsupervised DL model PCANet is introduced. Then, Section 3 describes the framework of KECANet and DAGSVM. Section 4 reports the performance of the method based on several experiments using facial expression images from multiple public datasets, including Extended Cohn-Kanade (CK+) database [26], Japanese Female Facial Expression (JAFFE) database [27], and CMU Multi-PIE face database [28]. Finally, main conclusions are given in Section 5.


## 2. Principal Component Analysis Network (PCANet)

PCANet is a simple unsupervised DL model based on the CNN for image recognition. In the PCANet [23], PCA replaces the convolution kernels of CNN to prevent learning filter kernels in each iteration like CNN. In addition, the network uses blockwise histograms to replace the pooling layer for down-sampling. Therefore, the PCANet model is simpler compared with some other convolutional network structure. Additionally, the parameter learning in PCANet requires no use of the backpropagation algorithm or pretraining convolution kernel through an automatic encoder network and deep confidence network. As a result, it saves plenty of computational complexity, and the training is highly effective.Figure1 shows the structure of the PCANet. It is comprised of two convolutional layers and one output layer. First, PCA filters are derived from a minimum reconstruction error for the input training sample. The filters convolve with the input image, and we input the convolved result into the second layer. Then, repeat the first layer in the second layer. Finally, hash coding and histogram block are applied to extract the ultimate features of images. The PCANet model relies on convolution to extract features, retaining spatial information of images. However, the method is restricted to the linear relation among the features, rather than the nonlinear relation when the image features are extracted. Therefore, the output image of PCANet will lose part of the original information. To resolve this problem, we propose the Kernel entropy component analysis network (KECANet) for image recognition in Section 3.Figure 1
The structure of the PCANet model.


## 3. Overview of the Proposed Algorithm

For each facial expression image, we first preprocessed the image. Then, the features of expression images are extracted by using the KECANet algorithm. The Figure2 shows the flowchart of facial expression recognition based on KECANet. Each face expression image has a uniform size after preprocessing. Next, in order to learn convolution kernels and obtain feature expression, the facial expression images were partitioned into training and testing sets.Figure 2
The flowchart of the FER based on KECANet.For more efficient feature extraction, features are extracted by using the proposed method inspired by PCANet in the stage of feature extraction. A series of mapping features are produced in the first stage, serving as input to the second stage. Then, the output in the second stage is used as the input of the output stage. Next, binary hashing encoding and block histogram were used to compute the final features. Finally, we sent the final output features to the DAGSVM classifier for facial expression recognition. Figure3 displays an illustration of KECANet.Figure 3
An illustration of the KECANet model.


### 3.1. Kernel Entropy Component Analysis (KECA)

KECA is a data feature extraction method based on information entropy [29, 30]. The method was first proposed by Robert Jenssen in 2010. KECA works by projecting raw data into higher-dimensional space to eigen decomposing the kernel matrix. Then, the eigenvector with the maximum eigenvalue is selected to form a new data space. It is underpinned by Renyi entropy and Parzen window. KECA can resolve the problem of the linear inseparability of the other model and enhances the separability between features. For the expression image training, sample X=x1,x2,...,xN, PX refers to the probability density function of X. The Renyi entropy can be written as Hp=−log∫p2xdx. The parzen window can be written as p^x=1/N∑xi⊂DKσx,xi, where x denotes the sample Vp=∫p2xdx. By approximating Vp with the Parzen window, we have(1)V^p=∫p2xdx=1N2∑i=1N∑j=1N∫Kσx,xiKσx,xjdx=1N2IKIT,where I denotes the N × 1 vector (every element is 1) and K denotes the N × N kernel matrix, Ki,j=Kσxi,xj. We map the high-dimensional feature spaces φ to subspaces Ud. Then, the reordering of eigenvalues and eigenvectors is made according to the entropy. The principal component matrix φeca can be obtained:(2)φeca=UdTφ=Dd1/2EdT,where X stands for a diagonal matrix composed of the first d eigenvalues of the matrix D and E represents a matrix with eigenvectors. The solution of the above equation is transformed into the solution of the minimum value problem, and we have(3)φeca=Dd1/2EdT:minλ1,α1,...,λd,αdV^p−V^dp,where V^dp=1/N2∑i=1dλiαiTI2. For a new sample xnew, the projection on the feature space can be expressed as follows:(4)W=UdTφxnew=Dd−1/2EdTφTφxnew=Dd−1/2EdTKx,xnew.


### 3.2. Features Extraction with the Proposed KECANet

#### 3.2.1. Image Reconstruction

Assume that we haveN1 input training images of Iii=1,2,3,...,N1. The size is m×n. According to Figure 4, patches of size k1×k2 are collected, where 0<k1≤m and 0<k2≤n. We collect all overlapping patches and vectorize them, i.e., Xi,1,Xi,2,...,Xi,m˜n˜∈Rk1k2, where m˜=m−k1+1 and n˜=n−k2+1. Next, the patch mean removal is done on all overlapping patches and Xi−=X−i,1,X−i,2,...,X−i,m˜n˜∈Rk1k2,i=1,2,...,N1 is obtained. In this paper, we define Xi−=X−i,1,X−i,2,...,X−i,m˜n˜∈Rk1k2,i=1,2,...,N1 as image reconstruction set (IRS).Figure 4
Illustration of 4 × 4 patch size used for 6 × 6 images.Then, we do IRS on all training samples and obtain(5)X=X1−,X−2,...,XN1−∈Rk1k2×N1mn.


#### 3.2.2. Image Convolution

The key to feature extraction is to study the mapping matrix. To learnk1×k2 convolution kernel, the KECANet is purposed to identify the eigenvectors and eigenvalues of the covariance matrix. Then, the optimal projection axis of the sample matrix is extracted by KECA as the convolution filter of the feature extraction stage Wl1, l=1,2,...,L1. L1 denotes the number of principal component eigenvectors of the first feature extraction stage. Wl1 is used as the filters of the first layer. Then, the boundary Ii is zero-padded and convolved with Wl1. Thus, we get the output of the first stage of feature extraction:(6)Oi,l=Ii∗Wl1,i=1,2,...,N1;l=1,2,...,L1.The identical mapping process conducted in the first stage is repeated in the second stage. The outputOi,l in the first stage serves as the input in the second layer. Then, KECA is used to getL2 convolution kernel WS2 of the second layer. So, the output of the second layer can be obtained after image convolution operation. The output can be expressed as(7)Oi,l,s=Oi,l∗Ws2,i=1,2,...,N:l=1,2,...,L1;s=1,2,...,L2.From the above,L1 matrices are produced from the first layer of KECANet. For every output matrix of the first layer, the second layer will produce L2 corresponding feature matrix. When the ith image enters the KECANet, L1L2 output matrix can be obtained.


#### 3.2.3. Output Stage

In the output stage, binary hashing encoding and block histogram are adopted, which can reduce the number of the feature matrix and produce more desirable classification result. The outputsOi,l,s are binarized to obtain the output HOi,l∗WS2. The binary vector is converted into a decimal number, and a single integer-valued image Til is obtained:(8)Til=∑l=1L22l−1HOil∗wS2.Each of the imagesTil computed by binary hashing is partitioned into B0<B<mn blocks. Then, the histogram of each block’s decimal values is computed. Next, all histograms are concatenated into the vector BhistTil,i=i,2,...,N1;j=1,2,...,L1. The feature vector fi of the input expression image is obtained through the encoding process:(9)fi=BhistTi1,...,BhistTiL1T.


### 3.3. DAGSVM

Based on the proposed KECANet model, we present an effective classifier DAGSVM. The DAGSVM is based on SVM to solve multiclass problem. Assuming that there areK facial expressions, the method constructs a direct acyclic graph containing K (K−1)/2 nodes and K leaves. Figure 5 shows a 4-layer DAGSVM classifier. For the ith and jth facial expression image, each node denotes a binary SVM classifier. Each leaf denotes a facial expression decision. The root node denotes the easiest classifier between two quite different expressions, whereas the node of the fourth stage denotes the most difficult classifier between two similar expressions.Figure 5
The architecture of 4-layer DAGSVM.The given test samplex is first put into the classifier of the root node to perform classification. The algorithm moves left or right for next computing based on the output value after the decision of the root classifier. The above steps are repeated until ending at the leaf node indicating the predicted facial expression.Because the numbers of different facial expression images are always unbalanced, we need to adjust the penalty parameters of different binary classifiers. A larger penalty parameter is set for a smaller number of subjects, and a smaller penalty parameter is set for a larger number of subjects.nk is the number of subjects, and ck is a penalty parameter of the kth facial expression image (k = 1, 2, ..., K). The penalty parameter is set as(10)ck=αnk,where α denotes the penalty coefficient common for all the facial expression images.


## 3.1. Kernel Entropy Component Analysis (KECA)

KECA is a data feature extraction method based on information entropy [29, 30]. The method was first proposed by Robert Jenssen in 2010. KECA works by projecting raw data into higher-dimensional space to eigen decomposing the kernel matrix. Then, the eigenvector with the maximum eigenvalue is selected to form a new data space. It is underpinned by Renyi entropy and Parzen window. KECA can resolve the problem of the linear inseparability of the other model and enhances the separability between features. For the expression image training, sample X=x1,x2,...,xN, PX refers to the probability density function of X. The Renyi entropy can be written as Hp=−log∫p2xdx. The parzen window can be written as p^x=1/N∑xi⊂DKσx,xi, where x denotes the sample Vp=∫p2xdx. By approximating Vp with the Parzen window, we have(1)V^p=∫p2xdx=1N2∑i=1N∑j=1N∫Kσx,xiKσx,xjdx=1N2IKIT,where I denotes the N × 1 vector (every element is 1) and K denotes the N × N kernel matrix, Ki,j=Kσxi,xj. We map the high-dimensional feature spaces φ to subspaces Ud. Then, the reordering of eigenvalues and eigenvectors is made according to the entropy. The principal component matrix φeca can be obtained:(2)φeca=UdTφ=Dd1/2EdT,where X stands for a diagonal matrix composed of the first d eigenvalues of the matrix D and E represents a matrix with eigenvectors. The solution of the above equation is transformed into the solution of the minimum value problem, and we have(3)φeca=Dd1/2EdT:minλ1,α1,...,λd,αdV^p−V^dp,where V^dp=1/N2∑i=1dλiαiTI2. For a new sample xnew, the projection on the feature space can be expressed as follows:(4)W=UdTφxnew=Dd−1/2EdTφTφxnew=Dd−1/2EdTKx,xnew.


## 3.2. Features Extraction with the Proposed KECANet

### 3.2.1. Image Reconstruction

Assume that we haveN1 input training images of Iii=1,2,3,...,N1. The size is m×n. According to Figure 4, patches of size k1×k2 are collected, where 0<k1≤m and 0<k2≤n. We collect all overlapping patches and vectorize them, i.e., Xi,1,Xi,2,...,Xi,m˜n˜∈Rk1k2, where m˜=m−k1+1 and n˜=n−k2+1. Next, the patch mean removal is done on all overlapping patches and Xi−=X−i,1,X−i,2,...,X−i,m˜n˜∈Rk1k2,i=1,2,...,N1 is obtained. In this paper, we define Xi−=X−i,1,X−i,2,...,X−i,m˜n˜∈Rk1k2,i=1,2,...,N1 as image reconstruction set (IRS).Figure 4
Illustration of 4 × 4 patch size used for 6 × 6 images.Then, we do IRS on all training samples and obtain(5)X=X1−,X−2,...,XN1−∈Rk1k2×N1mn.


### 3.2.2. Image Convolution

The key to feature extraction is to study the mapping matrix. To learnk1×k2 convolution kernel, the KECANet is purposed to identify the eigenvectors and eigenvalues of the covariance matrix. Then, the optimal projection axis of the sample matrix is extracted by KECA as the convolution filter of the feature extraction stage Wl1, l=1,2,...,L1. L1 denotes the number of principal component eigenvectors of the first feature extraction stage. Wl1 is used as the filters of the first layer. Then, the boundary Ii is zero-padded and convolved with Wl1. Thus, we get the output of the first stage of feature extraction:(6)Oi,l=Ii∗Wl1,i=1,2,...,N1;l=1,2,...,L1.The identical mapping process conducted in the first stage is repeated in the second stage. The outputOi,l in the first stage serves as the input in the second layer. Then, KECA is used to getL2 convolution kernel WS2 of the second layer. So, the output of the second layer can be obtained after image convolution operation. The output can be expressed as(7)Oi,l,s=Oi,l∗Ws2,i=1,2,...,N:l=1,2,...,L1;s=1,2,...,L2.From the above,L1 matrices are produced from the first layer of KECANet. For every output matrix of the first layer, the second layer will produce L2 corresponding feature matrix. When the ith image enters the KECANet, L1L2 output matrix can be obtained.


### 3.2.3. Output Stage

In the output stage, binary hashing encoding and block histogram are adopted, which can reduce the number of the feature matrix and produce more desirable classification result. The outputsOi,l,s are binarized to obtain the output HOi,l∗WS2. The binary vector is converted into a decimal number, and a single integer-valued image Til is obtained:(8)Til=∑l=1L22l−1HOil∗wS2.Each of the imagesTil computed by binary hashing is partitioned into B0<B<mn blocks. Then, the histogram of each block’s decimal values is computed. Next, all histograms are concatenated into the vector BhistTil,i=i,2,...,N1;j=1,2,...,L1. The feature vector fi of the input expression image is obtained through the encoding process:(9)fi=BhistTi1,...,BhistTiL1T.


## 3.2.1. Image Reconstruction

Assume that we haveN1 input training images of Iii=1,2,3,...,N1. The size is m×n. According to Figure 4, patches of size k1×k2 are collected, where 0<k1≤m and 0<k2≤n. We collect all overlapping patches and vectorize them, i.e., Xi,1,Xi,2,...,Xi,m˜n˜∈Rk1k2, where m˜=m−k1+1 and n˜=n−k2+1. Next, the patch mean removal is done on all overlapping patches and Xi−=X−i,1,X−i,2,...,X−i,m˜n˜∈Rk1k2,i=1,2,...,N1 is obtained. In this paper, we define Xi−=X−i,1,X−i,2,...,X−i,m˜n˜∈Rk1k2,i=1,2,...,N1 as image reconstruction set (IRS).Figure 4
Illustration of 4 × 4 patch size used for 6 × 6 images.Then, we do IRS on all training samples and obtain(5)X=X1−,X−2,...,XN1−∈Rk1k2×N1mn.


## 3.2.2. Image Convolution

The key to feature extraction is to study the mapping matrix. To learnk1×k2 convolution kernel, the KECANet is purposed to identify the eigenvectors and eigenvalues of the covariance matrix. Then, the optimal projection axis of the sample matrix is extracted by KECA as the convolution filter of the feature extraction stage Wl1, l=1,2,...,L1. L1 denotes the number of principal component eigenvectors of the first feature extraction stage. Wl1 is used as the filters of the first layer. Then, the boundary Ii is zero-padded and convolved with Wl1. Thus, we get the output of the first stage of feature extraction:(6)Oi,l=Ii∗Wl1,i=1,2,...,N1;l=1,2,...,L1.The identical mapping process conducted in the first stage is repeated in the second stage. The outputOi,l in the first stage serves as the input in the second layer. Then, KECA is used to getL2 convolution kernel WS2 of the second layer. So, the output of the second layer can be obtained after image convolution operation. The output can be expressed as(7)Oi,l,s=Oi,l∗Ws2,i=1,2,...,N:l=1,2,...,L1;s=1,2,...,L2.From the above,L1 matrices are produced from the first layer of KECANet. For every output matrix of the first layer, the second layer will produce L2 corresponding feature matrix. When the ith image enters the KECANet, L1L2 output matrix can be obtained.


## 3.2.3. Output Stage

In the output stage, binary hashing encoding and block histogram are adopted, which can reduce the number of the feature matrix and produce more desirable classification result. The outputsOi,l,s are binarized to obtain the output HOi,l∗WS2. The binary vector is converted into a decimal number, and a single integer-valued image Til is obtained:(8)Til=∑l=1L22l−1HOil∗wS2.Each of the imagesTil computed by binary hashing is partitioned into B0<B<mn blocks. Then, the histogram of each block’s decimal values is computed. Next, all histograms are concatenated into the vector BhistTil,i=i,2,...,N1;j=1,2,...,L1. The feature vector fi of the input expression image is obtained through the encoding process:(9)fi=BhistTi1,...,BhistTiL1T.


## 3.3. DAGSVM

Based on the proposed KECANet model, we present an effective classifier DAGSVM. The DAGSVM is based on SVM to solve multiclass problem. Assuming that there areK facial expressions, the method constructs a direct acyclic graph containing K (K−1)/2 nodes and K leaves. Figure 5 shows a 4-layer DAGSVM classifier. For the ith and jth facial expression image, each node denotes a binary SVM classifier. Each leaf denotes a facial expression decision. The root node denotes the easiest classifier between two quite different expressions, whereas the node of the fourth stage denotes the most difficult classifier between two similar expressions.Figure 5
The architecture of 4-layer DAGSVM.The given test samplex is first put into the classifier of the root node to perform classification. The algorithm moves left or right for next computing based on the output value after the decision of the root classifier. The above steps are repeated until ending at the leaf node indicating the predicted facial expression.Because the numbers of different facial expression images are always unbalanced, we need to adjust the penalty parameters of different binary classifiers. A larger penalty parameter is set for a smaller number of subjects, and a smaller penalty parameter is set for a larger number of subjects.nk is the number of subjects, and ck is a penalty parameter of the kth facial expression image (k = 1, 2, ..., K). The penalty parameter is set as(10)ck=αnk,where α denotes the penalty coefficient common for all the facial expression images.


## 4. Experiments and Results

In this section, we performed experiments on facial expression databases CK+, JAFFE, and CMU Multi-PIE to verify the performance of our proposed method. We split input the facial expression images into the size of 64 × 64 according to the two-eye positions. Then, we subjected the split expression images to down-sampling to 48 × 48 pixels.


### 4.1. Databases

In CK+ database, there are 593 facial expression image sequences from 123 subjects, including men and women aged 18–30 years from different countries. The dataset contains seven facial expressions including anger (135 images), sadness (28 images), surprise (246 images), disgust (177 images), fear (75 images), happiness (207 images), and neutral (314 images). Figure6 shows a proportion of exemplary images from the CK+ database.Figure 6
Sample facial images from the CK+ dataset.The JAFFE database consists of 213 facial expression images, including sadness (31 images), fear (32 images), surprise (30 images), anger (30 images), happy (31 images), disgust (29 images), and neutral (30 images). Figure7 presents some examples from JAFFE database.Figure 7
Sample facial images from the JAFFE dataset.In addition to abovementioned facial expression databases, we also evaluate the algorithm proposed in this paper on CMU Multi-PIE database. The dataset involves 337 subjects, obtained under 15 viewing angles and 19 lighting conditions for totally over 750,000 images. We selected 1,251 emoticons from 71 people for the experiment in this paper, containing six different expressions, including disgust, screaming, smile, squint, surprise, and neutral. Figure8 presents a proportion of expression image samples from the CMU Multi-PIE database.Figure 8
Partial samples of CMU Multi-PIE database.


### 4.2. Comparing the Average Accuracy of Different FER Methods

We compared the performance of KECANet with that of LDA, PCA, KPCA, as well as the LDANet and PCANet methods. The average recognition rates of different methods on three datasets are displayed in Table1.Table 1
Experiment results of different methods.
MethodsCK+JAFFECMU Multi-PIELDA73.2369.1770.15PCA72.2170.0272.19KPCA73.5671.2179.11PCANet90.8287.3290.72LDANet91.2987.2190.81KECANet92.6788.2191.19As shown in Table1, for the CK+ and JAFFE database, the average accuracy rates produced by our proposed method KECANet are above 90%. The KECANet method produces the accuracy rate of 92.67% especially on the CK+ dataset, whereas the average accuracies produced by other methods are below 92%. Likewise, our proposed method can reach an accuracy of 91.19% on CMU Multi-PIE database, whereas the accuracy rates are 90% or below by other methods. By comparing the above data, it is concluded that the performance of the method KECANet is superior to that of the other FER methods.Then, the proposed algorithm is tested on three datasets with the cross-validation method of Leaving one subject out (LOSO). We selected one subject out once as testing set and applied other subjects’ expression images for training set. In this verification method, there is no intersection between training and testing sets, which is close to the actual situation. Figures9–11 present the confusion matrices on three datasets in the LOSO scenario.Figure 9
The confusion matrices on CK+ dataset.Figure 10
The confusion matrices on JAFFE dataset.Figure 11
The confusion matrices on CMU Multi-PIE dataset.As revealed by the experimental data in Figures9 and 10, compared with other expressions, anger and sadness expressions are more likely to misclassified as neutral expression. The major causes include two aspects. (1) The expressions of anger and sadness have no exaggerated expression features. They are particularly close to neutral expressions, which make them easy to confuse. (2) The number of expressions of anger and sadness is smaller relative to other expressions in the CK+ dataset, for which they are unable to produce positive training results. Figure 11 shows that disgust and smile expressions are more likely to misclassified as neutral expression compared with other expressions.As revealed in Figures9 and 10, the recognition results of the proposed algorithm on CK+ and JAFFE are different, and the recognition rate on the CK+ dataset is greater than that on the JAFFE dataset. The leading cause is that the number of expression images of CK+ and JAFFE dataset is different, and the number of samples on the CK+ dataset is approximately twice as large as the JAFFE dataset.


### 4.3. Comparison of the Recognition Rates of Each Expression

For further validation of the effectiveness of our proposed method, Figures12–14 display the recognition results of each expression using different FER methods for three databases, and the performance of the proposed method precedes other FER methods in most cases.Figure 12
Comparing the test accuracies on the CK+ dataset.Figure 13
Comparing the test accuracies on the JAFFE dataset.Figure 14
Comparing the test accuracies on the CMU Multi-PIE dataset.


### 4.4. Effects of Filter

In this subsection, we investigate the influence of the number of filtersL1 for KECANet on CK+, JAFFE, and CMU Multi-PIE datasets. We fix L2 = 8 and change the filter numbers L1 of the first stage from 2 to 14. Figures 15–17 show the recognition results using PCANet, LDANet, and our proposed method KECANet with some filters in the first stage.Figure 15
The average recognition rate corresponding to different values ofL1 using CK + dataset.Figure 16
The average recognition rate corresponding to different values ofL1 using JAFFE dataset.Figure 17
The average recognition rate corresponding to different values ofL1 using CMU Multi-PIE dataset.As shown in Figures15–17, when the number of convolution in the first layer is comparatively low, PCANet and LDANet show a marginally greater recognition rate compared with KECANet. Nevertheless, with the growing number of convolution kernels in the first layer, the recognition rate of KECANet gradually exceeds PCANet and LDANet, thus showing an overall increasing trend.The experimental results indicate that the number of filterL1 varies with each algorithm with the highest recognition rate. Therefore, we set L1 in each algorithm to the number corresponding to its highest recognition rate. The allocation of the training sets and testing sets is identical to the prior experiment. In addition, to validate the robustness of each algorithm, we repeated each experiment 8 times. Tables 2–4 show the experimental results where L1 represents the filter numbers in the first stage and mean denotes the average recognition rate of 8 groups of experiments.Table 2
The recognition rate of CK+ database.
AlgorithmL1MeanPCANet1091.19LDANet1091.51KECANet892.57Table 3
The recognition rate of JAFFE database.
AlgorithmL1MeanPCANet1287.51LDANet1287.47KECANet888.35Table 4
The recognition rate of CMU Multi-PIE database.
AlgorithmL1MeanPCANet890.67LDANet890.81KECANet1091.22Tables2–4 show that the average recognition rate of the KECANet algorithm is noticeably greater than those of PCANet and LDANet.


## 4.1. Databases

In CK+ database, there are 593 facial expression image sequences from 123 subjects, including men and women aged 18–30 years from different countries. The dataset contains seven facial expressions including anger (135 images), sadness (28 images), surprise (246 images), disgust (177 images), fear (75 images), happiness (207 images), and neutral (314 images). Figure6 shows a proportion of exemplary images from the CK+ database.Figure 6
Sample facial images from the CK+ dataset.The JAFFE database consists of 213 facial expression images, including sadness (31 images), fear (32 images), surprise (30 images), anger (30 images), happy (31 images), disgust (29 images), and neutral (30 images). Figure7 presents some examples from JAFFE database.Figure 7
Sample facial images from the JAFFE dataset.In addition to abovementioned facial expression databases, we also evaluate the algorithm proposed in this paper on CMU Multi-PIE database. The dataset involves 337 subjects, obtained under 15 viewing angles and 19 lighting conditions for totally over 750,000 images. We selected 1,251 emoticons from 71 people for the experiment in this paper, containing six different expressions, including disgust, screaming, smile, squint, surprise, and neutral. Figure8 presents a proportion of expression image samples from the CMU Multi-PIE database.Figure 8
Partial samples of CMU Multi-PIE database.


## 4.2. Comparing the Average Accuracy of Different FER Methods

We compared the performance of KECANet with that of LDA, PCA, KPCA, as well as the LDANet and PCANet methods. The average recognition rates of different methods on three datasets are displayed in Table1.Table 1
Experiment results of different methods.
MethodsCK+JAFFECMU Multi-PIELDA73.2369.1770.15PCA72.2170.0272.19KPCA73.5671.2179.11PCANet90.8287.3290.72LDANet91.2987.2190.81KECANet92.6788.2191.19As shown in Table1, for the CK+ and JAFFE database, the average accuracy rates produced by our proposed method KECANet are above 90%. The KECANet method produces the accuracy rate of 92.67% especially on the CK+ dataset, whereas the average accuracies produced by other methods are below 92%. Likewise, our proposed method can reach an accuracy of 91.19% on CMU Multi-PIE database, whereas the accuracy rates are 90% or below by other methods. By comparing the above data, it is concluded that the performance of the method KECANet is superior to that of the other FER methods.Then, the proposed algorithm is tested on three datasets with the cross-validation method of Leaving one subject out (LOSO). We selected one subject out once as testing set and applied other subjects’ expression images for training set. In this verification method, there is no intersection between training and testing sets, which is close to the actual situation. Figures9–11 present the confusion matrices on three datasets in the LOSO scenario.Figure 9
The confusion matrices on CK+ dataset.Figure 10
The confusion matrices on JAFFE dataset.Figure 11
The confusion matrices on CMU Multi-PIE dataset.As revealed by the experimental data in Figures9 and 10, compared with other expressions, anger and sadness expressions are more likely to misclassified as neutral expression. The major causes include two aspects. (1) The expressions of anger and sadness have no exaggerated expression features. They are particularly close to neutral expressions, which make them easy to confuse. (2) The number of expressions of anger and sadness is smaller relative to other expressions in the CK+ dataset, for which they are unable to produce positive training results. Figure 11 shows that disgust and smile expressions are more likely to misclassified as neutral expression compared with other expressions.As revealed in Figures9 and 10, the recognition results of the proposed algorithm on CK+ and JAFFE are different, and the recognition rate on the CK+ dataset is greater than that on the JAFFE dataset. The leading cause is that the number of expression images of CK+ and JAFFE dataset is different, and the number of samples on the CK+ dataset is approximately twice as large as the JAFFE dataset.


## 4.3. Comparison of the Recognition Rates of Each Expression

For further validation of the effectiveness of our proposed method, Figures12–14 display the recognition results of each expression using different FER methods for three databases, and the performance of the proposed method precedes other FER methods in most cases.Figure 12
Comparing the test accuracies on the CK+ dataset.Figure 13
Comparing the test accuracies on the JAFFE dataset.Figure 14
Comparing the test accuracies on the CMU Multi-PIE dataset.


## 4.4. Effects of Filter

In this subsection, we investigate the influence of the number of filtersL1 for KECANet on CK+, JAFFE, and CMU Multi-PIE datasets. We fix L2 = 8 and change the filter numbers L1 of the first stage from 2 to 14. Figures 15–17 show the recognition results using PCANet, LDANet, and our proposed method KECANet with some filters in the first stage.Figure 15
The average recognition rate corresponding to different values ofL1 using CK + dataset.Figure 16
The average recognition rate corresponding to different values ofL1 using JAFFE dataset.Figure 17
The average recognition rate corresponding to different values ofL1 using CMU Multi-PIE dataset.As shown in Figures15–17, when the number of convolution in the first layer is comparatively low, PCANet and LDANet show a marginally greater recognition rate compared with KECANet. Nevertheless, with the growing number of convolution kernels in the first layer, the recognition rate of KECANet gradually exceeds PCANet and LDANet, thus showing an overall increasing trend.The experimental results indicate that the number of filterL1 varies with each algorithm with the highest recognition rate. Therefore, we set L1 in each algorithm to the number corresponding to its highest recognition rate. The allocation of the training sets and testing sets is identical to the prior experiment. In addition, to validate the robustness of each algorithm, we repeated each experiment 8 times. Tables 2–4 show the experimental results where L1 represents the filter numbers in the first stage and mean denotes the average recognition rate of 8 groups of experiments.Table 2
The recognition rate of CK+ database.
AlgorithmL1MeanPCANet1091.19LDANet1091.51KECANet892.57Table 3
The recognition rate of JAFFE database.
AlgorithmL1MeanPCANet1287.51LDANet1287.47KECANet888.35Table 4
The recognition rate of CMU Multi-PIE database.
AlgorithmL1MeanPCANet890.67LDANet890.81KECANet1091.22Tables2–4 show that the average recognition rate of the KECANet algorithm is noticeably greater than those of PCANet and LDANet.


## 5. Conclusions

In this paper, we propose a novel facial expression recognition method using the Kernel entropy component analysis network (KECANet) and DAGSVM. First, the deep features of training samples and testing samples are acquired based on the KECANet model. Secondly, the final features of images are extracted by the binary hashing coding and histogram block. Then, the features extracted from the KECANet model are classified by using the DAGSVM classifier. Finally, the experiments are carried out on facial expression databases CK+, JAFFE, and CMU Multi-PIE, and the experimental results verify the superiority of our proposed method.It is seen that our proposed method consolidates the idea of deep learning. It resolves the problem of the linear inseparability of the PCANet model and enhances the separability between features. Our future work will focus on how to reduce the runtime of the deep learning algorithm, integrate the high-level expression features, and apply the proposed method to other recognition problems.



---

*Source: 6616158-2021-01-05.xml*