A novel face recognition method based on fusion of LBP and HOG

IET Image ProcessingVolume 15, Issue 14 p. 3559-3572 ORIGINAL RESEARCH PAPEROpen Access A novel face recognition method based on fusion of LBP and HOG Ting Chen, School of Information Engineering, Chang'an University, Xi'an, Shaanxi Province, ChinaSearch for more papers by this authorTao Gao, Corresponding Author gtnwpu@126.com orcid.org/0000-0002-5750-7449 School of Information Engineering, Chang'an University, Xi'an, Shaanxi Province, China Correspondence Gao Tao, School of Information Engineering, Chang'an University, Middle-section of Nan'er Huan Road, Xi'an 710064, ShaanXi Province, China. Email: gtnwpu@126.com Li Shuying, School of Automation, Xi'an University of Posts & Telecommunications, Chang'an West St. Chang'an District, Xi'an 710121, ShaanXi Province, China. Email: angle_lisy@163.comSearch for more papers by this authorShuying Li, Corresponding Author angle_lisy@163.com School of Automation, Xi'an University of Posts & Telecommunications, Xi'an, Shaanxi Province, China Correspondence Gao Tao, School of Information Engineering, Chang'an University, Middle-section of Nan'er Huan Road, Xi'an 710064, ShaanXi Province, China. Email: gtnwpu@126.com Li Shuying, School of Automation, Xi'an University of Posts & Telecommunications, Chang'an West St. Chang'an District, Xi'an 710121, ShaanXi Province, China. Email: angle_lisy@163.comSearch for more papers by this authorXi Zhang, School of Information Engineering, Chang'an University, Xi'an, Shaanxi Province, ChinaSearch for more papers by this authorJinpei Cao, School of Information Engineering, Chang'an University, Xi'an, Shaanxi Province, ChinaSearch for more papers by this authorDachun Yao, School of Information Engineering, Chang'an University, Xi'an, Shaanxi Province, ChinaSearch for more papers by this authorYh Li, School of Information Engineering, Chang'an University, Xi'an, Shaanxi Province, ChinaSearch for more papers by this author Ting Chen, School of Information Engineering, Chang'an University, Xi'an, Shaanxi Province, ChinaSearch for more papers by this authorTao Gao, Corresponding Author gtnwpu@126.com orcid.org/0000-0002-5750-7449 School of Information Engineering, Chang'an University, Xi'an, Shaanxi Province, China Correspondence Gao Tao, School of Information Engineering, Chang'an University, Middle-section of Nan'er Huan Road, Xi'an 710064, ShaanXi Province, China. Email: gtnwpu@126.com Li Shuying, School of Automation, Xi'an University of Posts & Telecommunications, Chang'an West St. Chang'an District, Xi'an 710121, ShaanXi Province, China. Email: angle_lisy@163.comSearch for more papers by this authorShuying Li, Corresponding Author angle_lisy@163.com School of Automation, Xi'an University of Posts & Telecommunications, Xi'an, Shaanxi Province, China Correspondence Gao Tao, School of Information Engineering, Chang'an University, Middle-section of Nan'er Huan Road, Xi'an 710064, ShaanXi Province, China. Email: gtnwpu@126.com Li Shuying, School of Automation, Xi'an University of Posts & Telecommunications, Chang'an West St. Chang'an District, Xi'an 710121, ShaanXi Province, China. Email: angle_lisy@163.comSearch for more papers by this authorXi Zhang, School of Information Engineering, Chang'an University, Xi'an, Shaanxi Province, ChinaSearch for more papers by this authorJinpei Cao, School of Information Engineering, Chang'an University, Xi'an, Shaanxi Province, ChinaSearch for more papers by this authorDachun Yao, School of Information Engineering, Chang'an University, Xi'an, Shaanxi Province, ChinaSearch for more papers by this authorYh Li, School of Information Engineering, Chang'an University, Xi'an, Shaanxi Province, ChinaSearch for more papers by this author First published: 06 April 2021 https://doi.org/10.1049/ipr2.12192AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract As one of the hot topics in the field of computer vision research, face recognition technology has received significant attention due to its potentiality for a wide range of applications in government as well as commercial purposes. In practical applications, although several existing face recognition methods have achieved good performances in specific scenes, they easily suffer from a sharp decline in recognition rate if affected by different conditions of light, expression, posture and occlusion. Among many factors, influences of complex illuminations on face recognition are particularly significant. To further improve the performance of the existing local binary pattern (LBP) operator, neighbourhood weighted average LBP (NWALBP) is first proposed for fully considering the strong correlations between pixel pairs in the neighbourhood, which extends the traditional LBP uni-layer neighbourhood template window to the bi-layer neighbourhood template window and calculates the weighted average of bi-layer neighbourhood pixels in each direction. Then, inspired by center symmetric LBP (CS-LBP), centre symmetric NWALBP (CS-NWALBP) is further proposed, which can effectively reduce computation complexity by only comparing the weighted average values of the neighbourhood pixels that are symmetric about the centre pixel. Finally, by combining the merit of histogram of oriented gradient (HOG), a feature fusion algorithm named CS-NWALBP+HOG is suggested. Several experiments have eventually demonstrated that our proposed algorithms have more robust performance under complex illumination conditions if compared with many other latest algorithms. 1 INTRODUCTION As one of the hot topics in the field of computer vision research, face recognition technology has received significant attention due to its potentiality for a wide range of applications in government as well as commercial purposes, such as law enforcement, counter terrorism, border control, e-commerce, voter identification, banking etc. Through the efforts of several research and commercial institutions, many achievements have been made and face identification system can work very well under certain ideal acquisition conditions. However, its performance will be inevitably affected by many factors [1, 2], including lighting conditions, facial expressions, occlusions, clutters, pose variations, aging, background, skin color etc. Among these factors, face recognition tasks always encounter difficulties, especially under the influence of illumination complexities. In fact, if capturing the object images under good lighting conditions, it is relatively easy to obtain great identification performance. But the recognition results may become very poor if the object images are captured under uneven illumination, insufficient illumination, excessive illumination etc. The reason is that the difference between images from the same face sometimes is likely to be greater than the difference between images from different faces [3, 4]. Therefore, researchers put forward a series of improved solutions to overcome the above-mentioned problem, which can be divided into three categories, including image prepossessing algorithms, feature extraction algorithms based on spatial transformations, as well as feature extraction algorithms based on local features. The idea of image preprocessing algorithms is adopting various types of image enhancement methods before face recognition. There are four kinds of commonly used enhancement methods [5, 6]: grey level transformation with histogram equalisation as the representative [7-12], homomorphic filtering base on illumination reflectance model [13-18], light compensation based on Retinex theory [19-24], as well as image enhancement in gradient domain [25-30]. Grey level transformation based on global histogram equalisation can enhance image's overall contrast and is suitable for enhancing images which have lower overall grey value or dynamic range. Grey level transformation based on local histogram equalisation can enhance image's detail information and expand local greyscale range, but may lead to the block effect and lower calculation rate etc. Both homomorphic filtering and Retinex light compensation are frequency domain processing methods, and their computational efficiency are relatively high. Homomorphic filtering can be used to enhance images which have low local grey level value and high dynamic range, but it may cause over-enhancement phenomenon and suffer from poor performance in the highlight and shadow regions. Retinex theory is mainly used to estimate image brightness and compensate detail information in dark regions while maintaining the brightness of the whole image. However, Retinex light compensation inevitably adjusts some parameters for each image, and its processing capability in the brighter region is very poor. Light halos are easy to appear in the regions where there are large variations of the gradient values. Gradient domain enhancement can expand local gradient range while compressing overall dynamic range. Compared with other methods, it performs well in enhancing image shadow and highlight regions, but needs to reconstruct the image in gradient domain by certain numerical algorithms. In recent years, the above four kinds of methods are constantly improving. Histogram equalisation methods tend to combine with human vision, enhancing the image while preserving its original appearance. Homomorphic filtering and Retinex light compensation methods tend to combine with other methods, and their function of reducing noise can make up for the shortcomings of other methods. The gradient domain methods focus on the improvement of computational efficiency. However, there are still two main drawbacks for these image enhancement methods: (1) computational efficiency: although local histogram equalisation or gradient domain method work very well, their practical applications are limited by lacking fast computing algorithms, and are more suitable for image analysis without any real-time requirement; (2) application scope: all these image enhancement methods are based on the assumption that the greyscale value variations of the image are continuous and slow. However, when light reflections appear in certain regions, the grey level value variations of the image are great, and the histogram equalisation method become invalid because of this large dynamic range. If the grey level variations for one part of the image are continuous and slow, but for the other part are burst, the homomorphic filtering and Retinex light compensation methods will also fail to work. Although the gradient domain method can extract the burst location of the image, it is still difficult to avoid the influence of severe illumination variations on the image. As the second kind of the face recognition methods, feature extraction algorithms based on spatial transformations can transform image information of high-dimensional subspaces into image information of low-dimensional subspaces with more discriminating abilities, by utilising mathematical transformation theory to realise feature extraction. Feature extraction algorithms based on spatial transformations have shown their superior performance since image information of low-dimensional subspace are more robust to complex illumination variations. Two typical feature extraction algorithms are respectively principal component analysis (PCA) [31] and singular value decomposition (SVD) [32]. They first calculate the covariance matrix of the data set, then select a certain number of orthogonal bases to construct a projection subspace, which has relatively lower dimensions than that of original data set, and in the meanwhile can preserve the original information after subspace reconstruction to the maximum extent. 'Eigenface' can be obtained after dimension reduction, but it represents the global information of the face image and does not consider the local information, thus leads to lower recognition rate. Many improved algorithms based on PCA or SVD are proposed, such as kernel PCA (KPCA) [33], modular PCA (MPCA) [34], two-dimensional PCA (2DPCA) [35, 36], weighted subpattern PCA (Aw-SpPCA) [37], discriminative K-SVD (DK-SVD) [38], flustered SVD (FSVD) [39], but their performances are still not very good especially under complex illumination. Therefore, a series of improved algorithms are further proposed by combining with other methods, such as wavelet transform [40, 41], correlation filter [42], linear discriminant analysis (LDA) [43], retina modelling [44], virtual representation [45] etc. Moreover, in order to to utilise samples' high-order statistical characteristics, other algorithms are proposed, such as linear discriminant analysis (IDA) [46], discrete cosine transform (DCT) [47] etc. However, all algorithms mentioned above can only work well under the premises of sufficient available training samples and expensive calculating costs, which are indeed some inevitable obstacles for practical applications. More importantly, all of them are global feature extraction methods and can only obtain an image's overall information, thus they are very sensitive to local light and noise, and their capabilities to describe image's local texture details are barely satisfactory, especially under complex illumination variations. The main idea of the third kind of algorithm based on local feature extraction is as follows: dividing the whole image into several blocks and extracting key features for each block; then concatenating these corresponding features according to certain rules; finally, obtaining the eventual feature descriptions for the whole original image. Compared with the above-mentioned global feature extraction algorithms, local feature extraction algorithms always have stronger description abilities to image texture details and perform better robustness to illumination, noise, pose variations etc. Furthermore, among all the local feature extraction algorithms, local binary pattern (LBP) [48], histogram of oriented gradient (HOG) [49] and Weber local descriptor (WLD) [50] are the classical representative local descriptors since they can explore local illumination insensitive characteristics very well. The original LBP operator [48] is used to describe local image texture features by comparing the grey value of central pixel with each that of its eight surrounding neighbour pixels and calculating the image contrast within the templet of pixels 3*3. LBP operator is easy to be carried out and efficient, and it has the advantages of rotational invariance, as well as greyscale invariance, but it is difficult to meet various requirements of roughness and texture sizes for image identification in the practical applications. An improved circular LBP operator [51] is proposed by expanding the templet from pixels 3*3 with radius 1 to circular area with any radius value. However, it will lead to a significant increase in computation complexity. To solve the above problem, a further improved method called center symmetric LBP (CS-LBP) [51] is given and received extensive attention, which can significantly reduce the computational dimensions. Over the next years, many experts and scholars have proposed a variety of improved algorithms based on LBP, including Bayesian LBP (BLBP) [53], complete LBP (CLBP) [54], weighted LBP with adaptive threshold (W-LBPAT) [55], Multi-scale block LBP (MB-LBP) [56] etc, and all of them further enhance the description ability for texture details. HOG [57] is a kind of commonly used feature descriptor for computer vision, which can utilise local gradient histogram to describe local image region's edge and direction information very well. In view of the fact that HOG has strong robustness to light intensity and direction, several extended algorithms are given based on it. Some approaches based on HOG multi-feature fusion are proposed [58, 59], and the dimensions of holistic and local HOG features are reduced by using PCA or 2DPCA and LDA, which not only significantly raise the recognition rate and reduce the computing time but also has certain robustness to the influence of light. A method called layered fusion of CS-LBP and HOG feature is proposed to better capture image's edge information and thus improve recognition rate greatly [60]. Inspired by Weber's law, a simple yet very powerful and robust local descriptor called WLD is proposed based on the fact that human perception of a pattern depends not only on the change of a stimulus (such as sound, lighting) but also on the original intensity of the stimulus [50]. WLD consists of two components: differential excitation and orientation, and these two components construct a concatenated WLD histogram. It is revealed that WLD only utilises the texture information of the central pixel's surrounding pixels from the nearest neighbourhood layer, but the central pixel's excitation is also related to the texture information of its surrounding pixels from the outer neighbourhood layer, thus some key local texture information is not utilised sufficiently, leading to performance loss [61]. To solve the above problem, many optimal algorithms are suggested based on WLD. A local descriptor called Weber local binary pattern (WLBP) is proposed by effectively combining the advantages of WLD and LBP [62]. WLBP is a powerful texture descriptor including two components: differential excitation and LBP. The differential excitation extracts perception features by Weber's law, while LBP can describe local features splendidly. A method for face recognition using multi-scale WLDs and multi-level information fusion is presented [63]. Based on the local graph stucture (LGS) [64] and its improved symmetric-local LGS (SLGS) [65], a novel image feature representation descriptor called orthogonal symmetric local Weber graph structure (OSLWGS) is presented to further enhance recognition capability by fusing the merits of WLD and LGS [66]. There are also many other improved algorithms which can obtain stronger robustness to complex illumination and noise, such as local difference ternary sequences (LDT) [67], Weber synergistic center-surround pattern (WSCP) [68], local partial differential equation (L-PDE) [72] etc. Based on the classic deep learning net, many improved algorithms have been proposed. Some representative methods include VGGNet [73], ResNet [74], CNN+LBP [75], Spatial FCN [76], Crowd understanding [77, 78] and more, which easily suffer from the insufficient training sample. In practical applications, although the existing face recognition algorithms have achieved good results in specific scenes, they are easily affected by illumination, expression, posture and occlusion, leading to a sharp decline in recognition rate. Among many influencing factors, complex illumination has a particularly significant impact on face recognition. To obtain a good image description method which is insensitive to illumination, this paper proposes an improved fusion local feature extraction algorithm called CS-NWALBP+HOG. One important contribution of this paper is our proposed novel operator called CS-NWALBP, which not only smooths noise sensitivity of LBP operator but also reduces the original computational complexity, as well as strengthens the description ability for image gradient direction information. There are four steps in the implementation of CS-NWALBP: extending the original uni-layer neighbourhood model of LBP to an improved bi-layer neighbourhood model; then calculating the weighted average values of pixels from each direction in the bi-layer neighbourhood; next, comparing the weighted average values of pixels that are symmetrical near the central pixel; last, coding a series of comparing results according to certain rules. Moreover, a novel fusion algorithm is proposed by combining the merits of HOG and CS-NWALBP for further promoting a robust performance under illumination variations. Several experiments are conducted to verify the effectiveness of our proposed algorithm CS-NWALBP+HOG, and it has better performance compared with many other existing methods. The contributions of this paper are summarised as: To improve the effectiveness of the local binary pattern (LBP) operator, neighbourhood weighted average LBP (NWALBP) is first proposed by fully considering the strong correlations between pixel pairs in the neighbourhood. We propose one novel operator called CS-NWALBP, which not only smooths noise sensitivity of LBP operator but also reduces the original computational complexity. By utilising with the advantages of CS-NWALBP and HOG, a feature fusion algorithm named CS-NWALBP+HOG is suggested. This paper is organised as follows: Section 1 provides an overview of the background of face recognition, and gives comprehensive evaluations of the latest mainstream methods as well as our contributions. Section 2 introduces the original LBP and HOG. Section 3 describes our proposed NWALBP and CS-NWALBP in great detail, as well as presents a novel fusion algorithm based on HOG and CS-NWALBP to further overcome illumination sensitivities. Several experiments based on typical public databases are conducted and their corresponding performance comparisons are presented in Section 4. Section 5 provides a conclusion and suggests the future research works. 2 RELATED WORKS 2.1 LBP LBP proposed by Ojala et al. [48] as early as 1996 has been widely used for face recognition and showed its high discriminative power for texture classification under complex illumination. LBP is a non-parametric operator which is described as an ordered set of binary comparisons of pixel intensities between the centre pixel and its eight surrounding pixels. Figure 1 shows the original LBP template window with a size of 3*3, and the LBP code can be expressed as Equations (1) and (2): L B P P , R = ∑ i = 0 P − 1 s x i − x c 2 i (1) s x = 1 , if x ≥ 0 0 , if x < 0 (2)where x c corresponds to the grey value of a center pixel, x i to the grey value of the eight surrounding pixels. P represents the total number of the centre pixel's surrounding pixels and is equal to 8 for the original LBP operator. R is the radius of LPB template which is shown in Figure 1 and is equal to 1 for the original LBP operator. The calculation procedure of one LBP code example is illustrated in Figure 2. For any pixel, the original LBP operator only accounts for its relative relationship with its neighbour pixels, which makes itself insensitive to illumination intensities. FIGURE 1Open in figure viewerPowerPoint The original LBP template window with a size of 3 * 3 FIGURE 2Open in figure viewerPowerPoint The calculation procedure of one original LBP code example The original LBP operator extracts local features by utilising a template window with a size of 3*3, and it may confront with the problem that the range is too small to meet the needs of texture feature extraction with different sizes. Therefore, Ojala et al. extended the original LBP operator to circular LBP operator that can be with different sizes, and its calculation procedure is the same as that of the original LBP operator [51]. Several circular LBP operators with different values of P and R are shown in Figure 3. FIGURE 3Open in figure viewerPowerPoint Several circular LBP operators with different values of P and R The original LBP and its extended circular ones mainly compare local centre pixel's grey value with each grey value of its surrounding pixels and represent image's local features by their relative variations. Because of its simple calculation process, strong local description ability for texture details and good robustness to complex illuminations, LBP is widely used in image recognition and many other fields. However, if evaluating the effect of the original LBP and its extended circular ones, the grey values between the centre pixel and all of its surrounding neighbour pixels should be compared, and then rather long histograms may be produced but difficult to use in the context of a region. If the number of training sets is too large, it will further lead to the problem of slow system calculating operation and long feature extraction time. To address the problem, an improved LBP called CS-LBP was proposed which modified the scheme of comparing the pixels in the neighbourhood instead of comparing each pixel with the centre pixel [52]. The CS-LBP code can be expressed in Equations (3) and (4), and centre-symmetric pairs of pixels are compared as illustrated in Figure 4: CS − LBP = ∑ i = 0 P P 2 2 − 1 s x i − x i + P P 2 2 2 i , P = 8 (3) s x = 1 , if x ≥ 0 0 , if x < 0 (4)It can be seen in Figure 4 that for eight neighbours, LBP produces 256 (28) different binary patterns, whereas for CS-LBP this number is only 16 (24). Therefore, CS-LBP can effectively reduce the number of comparisons, and improve computational efficiency. Figure 5 shows example image 'LENA' and its corresponding feature images extracted respectively by LBP and CS-LBP. FIGURE 4Open in figure viewerPowerPoint Different comparing schemes of LBP and CS-LBP in the template window with a size of 3*3 FIGURE 5Open in figure viewerPowerPoint Example image 'LENA' and its corresponding feature images 2.2 HOG Histogram of oriented gradient (HOG) was initially described by Dalal et al. in the context of person detection [49], and now widely used in pedestrian detection and face recognition. In HOG, gradient histograms are used to extract local features, which can well describe the edge and direction information, and thus have strong robustness to lighting, direction as well as other factors. The implementation procedure of HOG is as follows: using a sliding window with fixed size to divide the target image into several blocks with certain overlaps; further dividing each block into small cells; then calculating the gradient direction and amplitude of the grey value for each pixel; and calculating the corresponding histogram for each cell; finally, concatenating these cell histograms from each block to form the final statistical histogram as the feature extraction image. Taking the target image with a size of 120*120 that is affected by illuminating variations as the example, feature extraction procedure of HOG is given as follows: first, carrying out greyscale transformation for the input target image, so as to reduce the interference from colour information when calculating the corresponding gradient values; second, utilising the sliding window with a size of 20*20 to obtain blocks; third, dividing each block into four cells uniformly and calculating the horizontal gradient G H as well as vertical gradient G V of pixel I ( x , y ) to obtain the marginal information, as given in Equations (5) and (6), thus, the corresponding gradient direction d ( x , y ) and the amplitude a ( x , y ) can be calculated as shown in Figures 7 and 8, respectively; fourth, dividing the range of [ 0 , 180 ∘ ] uniformly into 20 regions with nine gradient directions, and making statistics on features according to which region that each pixel from each cell belongs to; fifth, obtaining the block statistical histogram by concatenating each statistical histogram of cells and standardising the extracted features to relieve the influence of illumination variation, noise as well as other factors; finally, obtaining the entire statistical histogram features for the target image by concatenating each standardised histogram of blocks: G H x , y = I x + 1 , y − I x − 1 , y (5) G V x , y = I x , y + 1 − I x , y − 1 (6) d ( x , y ) = arctan G H ( x , y ) G V ( x , y ) (7) a x , y = G H x , y 2 + G V x , y 2 (8) The HOG operator is a description operator used to reflect the local gradient direction and gradient strength for a face image. The idea of HOG is utilising the gradient or edge direction distribution to reflect the edge or shape features for a target image. There are several merits as follows: (1) it is an efficient method for image edge acquisition which is particularly sensitive to image edge's shape information and gradient direction; (2) by dividing image into cells and doing local calculations, it is relatively easy to represent the relationship among local image pixels as well as match features in the process of image recognition; (3) it has good optical invariance as well as geometric invariance since HOG features are extracted based on small cells. 3 PROPOSED METHOD 3.1 Neighbourhood weighted average local binary pattern (NWALBP) Although there are several drawbacks, LBP and its extended operators still have a strong local description ability to image texture details for face recognition, and experts and scholars continue to improve them based on various fusion methods. Multi-texture CS-LBP [69] is proposed, which expands the LBP template window from the original size of 3 ∗ 3 to modified size of 9 ∗ 9 , then makes local average based on blocks to effectively solve the problem that single pixel is vulnerable to illumination variations. However, there are still two problems: (1) multi-texture CS-LBP adopts a larger size of template window and it is likely to lead to insufficient description to im