Face recognition algorithm based on feature descriptor and weighted linear sparse representation

IET Image ProcessingVolume 13, Issue 12 p. 2281-2293 Research ArticleFree Access Face recognition algorithm based on feature descriptor and weighted linear sparse representation Mengmeng Liao, Mengmeng Liao Department of Electronic Engineering, Fudan University, Shanghai, 200433 People's Republic of ChinaSearch for more papers by this authorXiaodong Gu, Corresponding Author Xiaodong Gu xdgu@fudan.edu.cn orcid.org/0000-0002-7096-1830 Department of Electronic Engineering, Fudan University, Shanghai, 200433 People's Republic of ChinaSearch for more papers by this author Mengmeng Liao, Mengmeng Liao Department of Electronic Engineering, Fudan University, Shanghai, 200433 People's Republic of ChinaSearch for more papers by this authorXiaodong Gu, Corresponding Author Xiaodong Gu xdgu@fudan.edu.cn orcid.org/0000-0002-7096-1830 Department of Electronic Engineering, Fudan University, Shanghai, 200433 People's Republic of ChinaSearch for more papers by this author First published: 17 September 2019 https://doi.org/10.1049/iet-ipr.2018.5263Citations: 4AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Generally, the commonly used sparse-based methods, such as sparse representation classifier, have achieved a good recognition result in face recognition. However, there exist several problems in those methods. First, those methods think that the importance of each atom is the same in representing other query samples. This is not reasonable because different atoms contain different amounts of information, their importance should be different when they together represent the query samples. Second, those methods cannot meet the real-time requirement when dealing with large data set. In this study, on the one hand, the authors propose a fast extended sparse-weighted representation classifier (FESWRC) by considering the different importance of atoms and using primal augmented Lagrangian method as well as principal component analysis. On the other hand, the authors propose a distinctive feature descriptor, named logarithmic-weighted sum (LWS) feature descriptor. The authors combine FESWRC and LWS and used for face recognition, this method is called face recognition algorithm based on feature descriptor and weighted linear sparse representation (FDWLSR). Experimental results show that FDWLSR can realise real-time recognition and the recognition rate can achieve 100.0, 100.0, 91.6, 93.4 and 87.4%, respectively, on the Yale, Olivetti Research Laboratory (ORL), faculdade de engenharia industrial (FEI), face recognition technology program (FERET) and labelled face in the wild datasets. 1 Introduction Face recognition is a hot topic in the fields of computer vision and pattern recognition, and many researchers have been working on it [1-4]. Various techniques have been proposed during the past few years. Methods based on features are often used for face recognition. Their main idea is to extract the features from the images and utilise the extracted features to recognise faces. For this kind of methods, the extracted features have a great influence on the recognition rate. Hence, it is very important to extract the features that have strong discriminative attributes. The facial features can be divided into global features and local features. The global features mainly describe the overall attributes of the face, such as skin colour and contour. However, the local features mainly describe the details of the face changes, such as characteristics of facial organs, some peculiar features of the face. According to the types of features, face recognition methods based on features can be roughly divided into global features methods, local features methods and mixed features methods. There exist many global features methods, such as principal component analysis (PCA) [5], linear discriminant analysis (LDA) [6], independent component analysis (ICA) [7] and texture features methods. PCA can be used to reduce the dimension of samples so that the recognition time is reduced, but its performance is poor when dealing with non-frontal face images. For the face recognition problem that involves occlusion and illumination variations, LDA does not handle very well. Compared with PCA, ICA is less affected by expression and illumination variations. Texture features methods have attracted much attention of researchers in recent years. There are many texture features methods, such as discrete wavelet transform (DWT) methods [8], S-transform (ST) [9] methods and discrete orthonormal ST (DOST) methods [10], but global features methods have several drawbacks. First, they are the single mode feature methods, and a non-linear epidemic structure in a subset of human face is difficult to find. Second, they are very sensitive to occlusion and illumination variations. Research has shown that local features methods [11, 12] are very promising in object recognition and uncontrolled face recognition. There are many kinds of local features methods, such as sub-pattern method [13] and Gabor wavelet methods [14]. Local preserving projections [15] is a kind of sub-pattern method, which not only can effectively solve the small sample problems of face recognition but also is robust to occlusion and light variations. Gabor wavelet filters, which could effectively extract the local features on multiple scales, have been successfully used in face recognition [16]. Compared with global features methods, such as LDA, Gabor wavelet methods are less sensitive to image changes, such as illumination variations and expression variations. However, local features methods neglect the relation between the local features. In the practical application, there is a great difference between the same face images which are taken in complex environment, thus the content and effect of the global features and local features are different. Hence, it is difficult to solve the complex face recognition problem by using single global features or local features. Thereby mixed features methods are proposed. In mixed features methods, global and local features are fused to realise the complementarity between two kinds of features. The statistical properties of image, such as covariance matrix and distribution characteristics, can also be regarded as features. They can reflect the characteristics of image, so the statistical properties of image can be used for face recognition. Sometimes logarithmic operation is also used for the extraction of features. Logarithmic image processing is a mathematical framework based on abstract linear mathematics [17]. It contains several specific algebraic and functional operations which can be used to manipulate image intensity values in a bounded range. These operations lead to new techniques to process images more accurately [18]. The logarithmic image processing replaces the linear arithmetic operations with a non-linear one which more accurately characterises the response of the human eye [19]. Thus, it can be used to improve the discriminability of features. For global features, local features, the statistical properties of image and logarithmic operation, only one to three of them are adopted in the above face recognition methods based on features, and in most cases two of them are used to construct a new feature, such as global features methods and local features methods, methods based on global features and logarithmic operation. In addition to the above methods, there exist other types of methods, such as extreme learning machine (ELM) [20] and sparse representation methods [21]. ELM is a single hidden layer network which can solve the face recognition problems very quickly, but its recognition rate is limited. Although sparse representation classifier (SRC) [21, 22] performs better than methods like PCA and LDA, it has no robustness for large continuous occlusion [23]. Yang et al. proposed a regularised robust coding (RRC) model [23] to improve the robustness of SRC, which could regress a given signal with regularised regression coefficients. By assuming that the coding residual and the coding coefficient are, respectively, independent and identically distributed, RRC seeks for a maximum posterior solution of the coding problem. However, it can only mitigate the occlusion in face recognition to some extent. In face recognition, the intra-class difference which is caused by variable expressions, illuminations, and disguises, can be shared across different subjects [24]. Based on this idea, Deng et al. proposed the extended sparse representation-based classifier (ESRC) [24]. ESRC uses the intra-class variant dictionary and gallery samples to represent the test samples, which improves the recognition performance for occlusion or non-occlusion face image. In ESRC, a test sample is represented by several atoms that come from the same class. For atoms within a same class, ESRC holds that they have equal importance when representing other samples together. Since different atoms even from the same class holds different amount of information or content, which suggests that they should be considered distinctively. Besides, traditional methods are used to solve the -min problems in ESRC, which is time consuming. Especially, when the dataset is very large, ESRC takes a lot of time to solve the -min problems, so it cannot meet the real-time requirement. In this paper, we propose a face recognition algorithm based on feature descriptor and weighted linear sparse representation (FDWLSR). In FDWLSR, we first define a logarithmic-weighted sum (LWS) feature descriptor which combines the advantages of global feature, local feature, statistical property of image and logarithmic operation. Next, we propose a representativeness estimator, and use it to estimate the representativeness of the atom. Then we propose a fast extended sparse-weighted representation classifier (FESWRC) by using the representativeness of the atom, primal augmented Lagrangian method (PALM) [25] and PCA. In FESWRC, on the one hand the representativeness of the atom is used to reflect the important of the atom when representing other samples. The representativeness of each atom is incorporated into the sparse representation process as a weight coefficient. On the other hand, PALM is used to solve the -min problems and PCA is used to reduce the dimensionality of dataset, which greatly reduce the running time. Finally, FDWLSR is obtained using FESWRC and LWS feature descriptor simultaneously. The main contributions of this paper are as follows: (i) Defined a LWS feature descriptor which describes images in a more accurate way. It has combined the advantages of DOST feature (global feature), Gabor feature (local feature), covariance (statistical property of image) and logarithmic operation. (ii) Propose a representativeness estimator which can estimate the expressing ability of the atom. (iii) Proposed the method of FESWRC that has shortened recognition time and improved recognition rate. On the one hand, FESWRC considers the importance of each atom in representing the query samples by assigning a weight to each atom, which improves the recognition rate. On the other hand, in FESWRC, PALM is used to speed up the solution of -min problems and PCA is used to reduce the dimension of samples, which greatly reduce the recognition time. The rest of the paper is organised as follows. Section 2 introduces LWS feature descriptor. Section 3 introduces proposed representativeness estimator. Section 4 is our proposed FESWRC. Section 5 is our face recognition algorithm based on FDWLSR. Section 6 gives experimental results. Section 7 is the conclusion. 2 LWS feature descriptor In this paper, we propose a LWS feature descriptor which combines those advantages of DOST feature, Gabor feature, covariance and logarithmic operation. Fig. 1 shows the construction process of LWS descriptor. In Fig. 1, and are the weights. Fig. 1Open in figure viewerPowerPoint Construction process of LWS feature descriptor 2.1 DOST feature Texture feature is a kind of global feature, which can describe the surface properties of the corresponding objects in an image. Not only is it invariant to the rotation transformation but also it has a strong ability to resist noise. There are many ways to measure texture feature, such as statistical methods and methods based on wavelet. ST [9] is closely related to the continuous wavelet transform using a complex Morlet mother wavelet [26], having advantages in preserving phase information and recovering the Fourier domain of an image [10]. However, there are a lot of redundancy in ST, which hinders the application of ST in texture analysis. Recent work shows that a discrete orthonormal basis can be used to accelerate the calculation of ST and eliminate the redundancy in the space-frequency domain [27]. Similar to DWT, DOST also provides a spatial frequency representation. Besides, DOST can preserve the phase information and allows for an arbitrary partitioning of the frequency domain, which achieve the zero redundancy. Furthermore, it can also achieve pixel-based texture description quickly. Based on those above characteristics of DOST, it has a wide range of applications in scientific research, such as texture classification [10] and signal analysis [27]. DOST is a pared-down version of the fully redundant ST [27]. The main idea of DOST is to form N orthogonal unit-length basis vectors. Each base vector corresponds to a specific region in the time–frequency domain. Each specific region can be determined by the following three parameters: , and . is the centre of each frequency domain band, is the width of that band, is the location in time domain. For the setting of the values of , and , refer [27]. Thus the th basis vector can be defined by (1) (1)where . For any one-dimensional signal , its length is N. Its DOST coefficients for the region corresponding to the choice of can be obtained by (2) (2) can be written as the form in (3) (3)where the content in the square brackets is . Thus, (3) can be written as (4) (4)For each input image, the DOST coefficient matrix can be obtained by DOST, and the DOST coefficient matrix has the same size of the input image. 2.2 Gabor wavelet feature Since the coefficient of wavelet transform has a good visual characteristic and biological background, it is widely used in image processing, pattern recognition and other fields. Two-dimensional Gabor wavelet transform is a kind of wavelet transform, it not only has the above good properties but also has a good time–frequency localisation characteristic. In addition, the function of two-dimensional Gabor wavelet is similar to enhancing bottom image features including edge and peak, as well as local features; and its wavelet characteristics show that Gabor filter can describe the local grey level distribution well. The extracted Gabor feature not only has a good spatial locality and directional selectivity but also is robust to illumination and pose variations. Hence, in this paper, we use the two-dimensional Gabor wavelet to extract the local feature of image. For each pixel in an image, a vector can be obtained by (5)where represents the intensity value of position , is the value of a two-dimensional Gabor wavelet centred at with orientation and scale : (6)where and . For an image Q, we assume that the row number of Q is A, the column number of Q is B. Its Gabor wavelet feature is denoted by (7) 2.3 LWS feature descriptor For image Q, its Gabor wavelet feature is . The covariance matrix of is denoted by . Meanwhile, image with the size of is obtained by down-sampling the image Q. The DOST coefficient matrix of image is obtained by (4). The covariance matrix of is denoted by . Taking image Q as an example, we define a new LWS feature descriptor (8)where is the LWS feature descriptor of Q, represents the logarithmic operation. Here, we let , , . Then (8) can be written as (9)The LWS feature descriptors of other images in data set are obtained in this way. 3 Representativeness estimator In sparse representation methods, a test sample is represented by several atoms. For different atoms, because they contain different amounts of information, they have different representativeness to represent other images generally. The concept of information entropy describes how much information is provided by the signal or image [28, 29]. The larger the information entropy of the image is, the more information it contains [30]. However, the adjacent pixel values of the image are highly correlated, so that the image contains redundant information, which is not conducive to calculating the amount of information contained in the image (the form of the image is a matrix) [31]. Hence, we can first use the whitening to remove the redundant information of the image, and then calculate the amount of information of the image. Base on the above content, in this paper, we propose the representativeness estimator, and use it to estimate the representativeness of the atom. The main steps to estimate the representativeness (expressing ability) by representativeness estimator are as follows. First, we remove the relativity of the sample by whitening. Then its representativeness is obtained by computing its information entropy. Given an arbitrary sample , we assume that its size is . 3.1 Remove the relativity of sample (i) Remove the mean of (10)where is the mean of . ii) Compute the auto-covariance matrix (11)where represents the mathematical expectation. iii) Singular value decomposition (12)where is a diagonal matrix. iv) Compute the whitening matrix (13) (v) Compute the whitened matrix (14) 3.2 Compute the information entropy (i) is arranged into a matrix , and the size of is . (ii) Compute the information entropy of (15)where is the information entropy of and represents the proportion of pixel whose greyscale is i. In this paper, represents the representativeness of sample ; and the greater the value of , the stronger its representativeness. 4 Fast extended sparse-weighted representation classifier 4.1 Extended sparse representation-based classifier The idea of ESRC is as follows. The intra-class variability, which was caused by variable expressions, illuminations, and disguises, can be shared across different subjects. The possible variation between the atoms and test samples can be represented by constructing an intra-class variant dictionary. Hence, test samples can be represented by gallery samples and intra class variant bases. We assume that is the gallery sample (atom), where is the training sample with class i. Each column of represents a training sample with class i. That is to say, contains training samples with class i. Let represents the intra-class variant base (atom), where , is the class centroid of class i, , , . is the training sample, and it is used to obtain the intra-class variant base. Each column of represents a training sample with class i. That is to say, contains training samples. For any test sample , it can be expressed as (16)where and are the sparse vectors, whose entries are zeros except those associated with the ith class. z is a noise term with bounded energy , and . Equation (16) can be rewritten as (17) (17)where , . , . The residual is computed by (18)where , is a new vector whose only non-zero entries are the entries in those are associated with class i. The category of the atoms which make obtain the minimum value is the category of probe sample . 4.2 Fast extended sparse-weighted representation classifier (FESWRC) In ESRC, for atoms with the same class, because they contain different amounts of information, they have different representativeness to represent other samples. That is to say, their importance is not the same when they together represent other samples. In order to reflect their importance, we propose a FESWRC. Fig. 2 is the diagram of FESWRC. Fig. 2Open in figure viewerPowerPoint Weighted representation diagram of FESWRC The main idea of FESWRC also can be illustrated by (19) (19)where and are the weight coefficient vectors, which correspond to and , respectively. is the Hadamard product operator (20) (21)where corresponds to , . corresponds to , . In our FESWRC, , . For those samples with class i, the expression of is as follows: (22)where represents the contribution factor of jth sample in class i(23)where represents the information entropy of jth sample in class i. In ESRC, traditional solution methods, such as homotopy method [32], proximal point methods [33-35], approximate message passing (AMP) [36], templates for convex cone solvers [37] and parallel coordinate descent [38], are used to solve -minimisation problems. However, this is time consuming. It has been proved that PALM [25] is faster than other algorithms in solving -minimisation problems. Hence, we use PALM to solve the -minimisation problems in our FESWRC algorithm. Besides, PCA [39] is used to reduce the dimension of data set in FESWRC. Let , where represents the new sample (atom) with class i. Each column of represents a new sample. , where , is the class centroid of class i, , , . is the new training sample, and it is used to obtain the intra-class variant base (atom). Each column of represents a new training sample with class i. That is to say, contains new training samples. The dimensions of , and are reduced by PCA. Then, we can obtain , and , where , and correspond to , and , respectively. Let , , , . For the following sparse representation problem: (24)where is the test sample that needs to be represented and is the sparse dictionary. Then (24) can be written as the (25) by adding the quadratic penalty term (25) The minimisation problem in (25) can be solved by the augmented Lagrangian method (26) where is the augmented Lagrangian function, is a positive and is a Lagrange multiplier. Equation (27) gives a basic iterative scheme which can be used to solve (26) (27) We assume that the calculated value of is . Then, the residual can be computed by (28)where , is a new vector whose only non-zero entries are the entries in those are associated with class i. The category of probe sample can be obtained by (29)Hence, the label of is . Fig. 3 is the flow chart of FESWRC (Fig. 4). Fig. 3Open in figure viewerPowerPoint Flow chart of FESWRC Fig. 4Open in figure viewerPowerPoint Fast extended sparse-weighted representation classifier 5 Face recognition algorithm based on FDWLSR On the one hand, the defined LWS feature descriptor is robust to illumination variations and pose variations. On the other hand, FEWSRC not only has a strong robustness to occlusion and non-occlusion but also can meet the real-time requirement. In order to combine the advantages of LWS feature descriptor and FEWSRC, we propose a face recognition algorithm based on FDWLSR. The FDWLSR is as follows. First, LWS feature descriptor of each image is obtained by (9). Then the obtained LWS feature descriptors are used as the input of FESWRC. Finally, the labels of testing samples are obtained. Fig. 5 is the flow chart of FDWLSR. Fig. 5Open in figure viewerPowerPoint Flow chart of FDWLSR 6 Experimental results and analysis Our FDWLSR is compared with other state-of-the-art algorithms on Yale, ORL, FEI, FERET and labelled face in the wild (LFW) datasets, respectively. The MATLAB code of our FDWLSR will be publicly available at our personal homepage. 6.1 Yale dataset The Yale dataset contains 15 different person face images, and there are 11 different images (centre-light, wear glasses, happy, left-light, no wear glasses, normal, right-light, sad, sleepy, surprised and wink) about each person. The size of each image is . For each person, we randomly select eight images as the training set, and the rest images are used as the test set. In our experiments, each image are down sampled to . Some samples of Yale dataset are shown in Fig. 6a. Fig. 6Open in figure viewerPowerPoint Samples of different datasets (a) Some samples of Yale dataset, (b) Some samples of ORL dataset, (c) Some samples of FEI dataset, (d) Some samples of FERET dataset, (e) Some samples of LFW dataset Table 1 lists therecognition rates and recognition time of different algorithms on Yale dataset. Table 1. Recognition rates and recognition time of different algorithms onYale dataset Algorithm Recognition rate,% Recognition time,s RRC [23] 98.0 10.00 ESRC [24] 92.0 5.95 SRC [21] 92.0 5.15 SSRC [40] 94.5 37.68 SSRC [41] 93.9 16.36 S3RC [42] 95.2 36.97 SDR [43] 96.4 45.81 KNN [44] 98.0 3.06 NN [45] 98.0 3.91 ELM [20] 94.0 1.90 ELMSRC [46] 98.0 2.18 RSC [47] 96.0 40.01 SPDSRC [48] 86.0 11.46 KSRC [49] 98.0 38.33 FDWLSR 100.0 7.95 As shown in Table 1, the recognition rate of our FDWLSR has achieved a perfect 100.0%, which outperforms all the other algorithms listed. The recognition rate of FDWLSR is ∼14, 8, 8, 6, 6, 6, 5, 4 and 4% higher than those of SPDSRC, ESRC, SRC, SSRC [40], SSRC [41], ELM, S3RC, RSC and SDR, respectively. The recognition time of FDWLSR is 7.95 s, ∼1/5 that of RSC, that of KSRC, that of SSRC [40], that of SSRC [41] and that of SDR. Although the ELM consumes less time than FDWLSR, its recognition rate is 6% lower than that of FDWLSR. 6.2 ORL dataset The ORL dataset contains 40 distinct subjects face images, and there are 10 different images in each subject. For some subjects, the images were taken at different time, varying the lighting, facial expressions (open/closed eyes, smiling/not smiling) and facial details (glasses/no glasses). The size of each image is . For each subject, we randomly select seven images as the training set, and the rest images are used as the test set. In our experiments, all images are down sampled to . Some samples of ORL dataset are shown in Fig. 6b. Table 2 lists the recognition rates and recognition time of different algorithms on ORL dataset. Table 2. Recognition rates and recognition time of different algorithms on ORL dataset Algorithm Recognition rate, % Recognition time, s RRC [23] 93.3 87.99 ESRC [24] 92.5 54.92 SRC [21] 92.5 151.20 SSRC [40] 95.1 157.32 SSRC [41] 93.6 62.37 S3RC [42] 95.3 136.95 SDR [43] 96.7 182.48 KNN [44] 95.8 2.98 NN [45] 95.8 0.79 ELM [20] 91.7 4.58 ELMSRC [46] 93.3 52.10 RSC [47] 82.5 130.23 KSRC [49] 90.8 144.36 FDWLSR 100.0 25.12 Table 2 shows that the recognition rate of our FDWLSR is again 100.0%, with at least 17, 9, 8, 7, 7, 6, 6, 6, 4, 4 and 3% improvements compared to RSC, KSRC, ELM, ESRC, SRC, ELMSRC, RRC, SSRC [41], SSRC [40], S3RC and SDR, respectively. On the one hand, our FDWLSR is based on the sparse representation method, thus our FDWLSR mainly compares the running time against that of the other sparse representation methods. The recognition time of FDWLSR is 25.12 s, which is ∼2/7 that of RRC, that of ESRC, that of SRC, that of RSC, that of SSRC [40], that of SSRC [41], that of SDR and that of KSRC. On the other hand, although the recognition time of FDWLSR is longer than that of KNN, NN and ELM, the recognition rate of FDWLSR is a few percentage points higher than their recognition rates. 6.3 FEI dataset The FEI dataset is a Brazilian face dataset, and it contains 200 different persons. Each person has 14 different images. All images are colourful and taken against a white homogenous background in an upright frontal position with profile rotation of up to ∼180 degrees. Scale might vary ∼10% and the original size of each image is pixels. In our experiments, the whole dataset is used for our experiments. The training set and test set both account for half of the whole dataset. That is to say, for each person, we randomly select seven images as the training set, and the rest seven images are used as the test set. In our experiments, all images are down sampled to . Fig. 6c shows some examples of FEI dataset. Table 3 lists the recognition rates and recognition time of different algorithms on FEI dataset. Table 3. Recognition rates and recognition time of different algorithms on FEI dataset Algorithm Recognition rate, % Recognition time, s RRC [23] 56.4 1630.42 ESRC [24] 79.1 1397.48 SRC [21] 79.1 2805.54 SSRC [40] 83.4 2671.18 SSRC [41] 82.5 1022.47 S3RC [42] 84.3 2473.92 SDR [43] 86.4 2948.76 KNN [44] 66.1 51.32 NN [45] 66.1 106.31 ELM [20] 61.5 41.77 ELMSRC [46] 80.7 1630.02 RSC [47] 71.3 2796.31 KSRC [49] 77.9 2562.37 FDWLSR 91.6 465.99 From Table 3, the recognition rate of our FDWLSR is 91.6%, which achieves the highest recognition rate, with ∼35, 30, 25, 20, 14, 12, 12, 11, 9, 8, 7 and 5% improvements compared to RRC, ELM, KNN, RSC, KSRC, ESRC, SRC, ELMSRC, SSRC [41], SSRC [40], S3RC and SDR. The recognition time of FDWLSR is 465.99 s, which is ∼2/7 that of RRC, that of ESRC