Single sample description based on Gabor fusion

IET Image ProcessingVolume 13, Issue 14 p. 2840-2849 Special Section: Adversarial Learning in Image ProcessingFree Access Single sample description based on Gabor fusion Ting Chen, Ting Chen School of Information Engineering, Chang'an University, Middle-section of Nan'er Huan Road, Xi'an, Shaanxi Province, People's Republic of ChinaSearch for more papers by this authorTao Gao, Corresponding Author Tao Gao gtnwpu@126.com orcid.org/0000-0003-0325-7912 School of Information Engineering, Chang'an University, Middle-section of Nan'er Huan Road, Xi'an, Shaanxi Province, People's Republic of ChinaSearch for more papers by this authorXiangmo Zhao, Xiangmo Zhao School of Information Engineering, Chang'an University, Middle-section of Nan'er Huan Road, Xi'an, Shaanxi Province, People's Republic of ChinaSearch for more papers by this author Ting Chen, Ting Chen School of Information Engineering, Chang'an University, Middle-section of Nan'er Huan Road, Xi'an, Shaanxi Province, People's Republic of ChinaSearch for more papers by this authorTao Gao, Corresponding Author Tao Gao gtnwpu@126.com orcid.org/0000-0003-0325-7912 School of Information Engineering, Chang'an University, Middle-section of Nan'er Huan Road, Xi'an, Shaanxi Province, People's Republic of ChinaSearch for more papers by this authorXiangmo Zhao, Xiangmo Zhao School of Information Engineering, Chang'an University, Middle-section of Nan'er Huan Road, Xi'an, Shaanxi Province, People's Republic of ChinaSearch for more papers by this author First published: 01 November 2019 https://doi.org/10.1049/iet-ipr.2018.6665Citations: 3AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Owing to lack of enough face image and invalidation of many traditional face recognition algorithms, face recognition with single training sample is really a great challenge. To solve the above problem, this study proposes a novel local weighted fusion Gabor (LWFG) algorithm. First, one single sample is segmented into a series of block sub-images, and then, each of these sub-images is decomposed into a series of multi-resolution Gabor wavelets with multi-orientation and multi-scale. Second, different orientation Gabor wavelets with the same scale are fused. Next, different scale Gabor wavelets with the same orientation are fused according to the proposed fusion criterion. Third, the fusion Gabor feature histograms are calculated in each of the divided local regions. Meanwhile, every local region's information importance is measured by the proposed local image information content model. Finally, the fusion Gabor wavelet histograms are adaptively weighed by weighting map which calculated from information content model. This study conducted simulation experiments on different face databases under the different conditions including partial occlusion, expression change and illumination variation. The results indicated that the proposed LWFG algorithm is more effective with single training sample. 1 Introduction In recent years, face recognition has become an extensive topic in the field of image processing which has obtained several remarkable achievements. As a key step for face recognition, there are many face feature extraction algorithms. Shin et al. [1], Dubey et al. [2] and Wang et al. [3] proposed some face feature description algorithms using singular value decomposition. Fàbregas et al. brought linear discriminant analysis (LDA) into pattern recognition field when extracting the discriminant feature [4-14]. Some extension LDA algorithms have been studied and applied to face recognition such as the combined LDA with principal component analysis (PCA) and minimum distance classifier [5], gradient LDA [6], hash LDA [7], LP-norm LDA [8]. Shu et al. present some improved algorithms based on basic LDA method [9-14]. Toutanova and Johnson [15] and Newman and Block [16] proposed a novel face feature extraction method using topic decomposition model. Wang et al. [17-19] presented some feature selection methods to extract feature. Wang et al. utilised Gabor wavelets for face information feature modelling [20-27]. Kirby and Sirovich [28] first presented the idea of eigenface using PCA algorithm to extract face feature in 1990. Some extensions of PCA have been studied and applied to face recognition: block PCA [29, 30], two-dimensional (2D) PCA [31, 32], incremental PCA [33]. Selvan et al. presented the independent component analysis for face feature extraction [34-38]. The above algorithms can always work very well when there have been sufficient training samples. However, performance of those recognition algorithms will deteriorate significantly if the number of training face image is only one for each person. For one training face sample, the commonly used subspace analyses are neither no longer suitable nor applicable [39]. The single sample per person recognition is a main problem and a great challenge in practical application [40, 41]. Therefore, the single training sample problem brings more challenges to face recognition and is to be worth paying more attention to [42, 43]. Many remarkable algorithms have been proposed: sample enhancement algorithm [44], sample visual expansion algorithm [40, 45-47], general learning framework algorithm [48, 49], feature subspace algorithm [26, 50-55] etc. Some single face sample feature extraction algorithms have been studied and applied to face recognition: multi-directional orthogonal gradient phase [45], multi-resolution feature fusion [40], super-resolution subspace projection [46], joint kernel regression and adaptive dictionary learning [47]. Some improved local feature extraction algorithms also have been applied to single face recognition: local binary pattern (LBP) pyramid [50, 51], local ternary patterns (LTPs) [52], local structure-based image decomposition [53], fusion of local normalisation and Gabor entropy weighted features [26], local difference binary [50], scale-adaptive directional and textural features [55, 56], Weber synergistic centre-surround pattern [57], orthogonal symmetric local Weber graph structure [58]. There are also some Gabor feature extraction algorithms [59-61] applied to single face recognition: local Gabor (LG) algorithm [59], Gabor occlusion dictionary [60] and Gabor feature hallucination [61]. The fusion algorithms of Gabor and other local feature extraction were also proposed: local normalisation entropy-such as weighted Gabor features [26], LG binary patterns (LGBPs) and LG exclusive OR (XOR) patterns [62], Gabor wavelets and LBP [63], Gabor wavelets combined with volumetric fractal dimension [64], combined LBP, local phase quantisation and fuses Gabor filters [65], LG XOR patterns [66]. Since the computation amount of Gabor frames is really high, some accelerated Gabor methods have been studied: accelerated Gabor frames [67] and fusion of multi-channels classifier [68]. From the foregoing, face feature extraction algorithms with single training sample have made great achievements in face recognition. However, each kind of those algorithms has its own flaws. Some sample visual expansion algorithms make the traditional matrix decomposition method become available, but they have serious shortcomings in local texture description and are especially sensitive to expression changes. Some general learning framework algorithms cannot work very well due to lack of general complete face database. There are also some disadvantages when using feature subspace algorithms, since these algorithms cannot comprehensively utilise the importance of local region information for the whole face sample, and the results will often be affected by complex illumination, partial occlusion and different facial expression. To solve these above existing problems, this paper proposes a novel local weighted fusion Gabor (LWFG) algorithm which can make single training face feature extraction with adaptive weighted fusion of multi-orientation and multi-scale Gabor features. The proposed LWFG algorithm can fuse the different orientation Gabor wavelets which corresponding to the same scale and the different scale Gabor wavelets which corresponding to the same orientation based on the proposed fusion criterion. Furthermore, the fusion Gabor wavelet histograms are adaptively weighed by weighting map which is calculated by local image information content model. The contributions of the proposed LWFG algorithm are summarised as follows: (i) Fusion Gabor feature: The proposed LWFG algorithm fuses different scales and different orientations according to the proposed fusion criterion. The obtained performance of fusion Gabor feature is better than that of ordinary Gabor feature which itself also has superior performance for image feature extraction. Moreover, fusion Gabor feature histograms can be calculated to extract richer discriminative feature. (ii) Local image information content model: Psychological studies reveal that some facial regions such as eyes and mouth always play more important roles for face recognition than other facial regions. However, traditional local descriptors usually view the importance of every local region as the same which will make the complete global feature of one image cannot be truly reflected by the connected local features. To solve the above-mentioned problem, adaptive weighted fusion Gabor feature is introduced into the proposed local image information content model which can describe the global feature of one image more effectively. (iii) LWFG algorithm based on multi-degree block model and LDA: In the real world, there is usually only one sample image available for each person in some face recognition applications such as law enforcement, driver license or passport card's identifications, which leads to one image per person problem. In this condition, face recognition will suffer from serious performance degradation or even be unable to realise if adopting most of the existing traditional algorithms which require multiple gallery images for each person to train the system with optimal settings. By contrast, the proposed LWFG algorithms use multi-degree block model and LDA reduction to improve recognition efficiency and thus can work very well under special conditions of partial occlusion, complex illumination, different expressions when only one single sample image for enrolled subject is available. This paper is organised as follows: Section 1 introduces face recognition background and this paper's contributions. Section 2 presents implementation process of the proposed LWFG algorithm. Experiments and analysis based on typical face databases such as olivetti research laboratory (ORL), Yale, Yale B, carnegie mellon university pose-illumination-and-expression (CMUPIE) and face recognition technology (FRET) are conducted to compare recognition performance of the proposed algorithm with other traditional ones in Section 3. Section 4 makes conclusions and introduces the future work. 2 Algorithm description Framework of the proposed single training face feature extraction with adaptive weighted fusion of multi-orientation and multi-scale Gabor features is shown in Fig. 1. The input images are divided into two categories: training images and testing images. Since face feature extraction procedures for those two categories are the same, this paper only gives the training image's feature extraction process. First, the single training sample is segmented into a series of block sub-images according to the multi-scale partition model. Second, the process is divided into two branches: (i) for one thing, the sub-image is decomposed into multi-resolution Gabor wavelets with multi-orientation and multi-scale, and those Gabor wavelets with different orientation and scale are furthermore fused according to the proposed fusion criterion. Then, the fusion Gabor feature histograms are calculated in every divided local region; (ii) for another thing, the information importance of every local region for the whole face sample is measured by the proposed local image information content model. Third, the fusion Gabor wavelet histograms are adaptively weighed by weighting map which is calculated from the information content model, and finally reduced by LDA and recognised by radial basis function (RBF). The above-mentioned implementation process is illustrated in Fig. 2 and its corresponding LWFG algorithm is depicted as follows. Fig. 1Open in figure viewerPowerPoint Framework of the proposed single training face feature extraction Fig. 2Open in figure viewerPowerPoint Implementation process of the proposed single training face feature extraction 2.1 Multi-scale image partition If multi-channel Gabor filters are directly applied to one face image, the whole outline feature of this image can be well-represented. However, some local texture features cannot be characterised very well such as eyes, mouth, nose etc. Suppose the sample image is X with the size of and was divided as a matrix shown in the equation below: (1)where denotes the (i, j)th sub-image block with the size of . 2.2 Extraction of LG wavelets Each channel Gabor filter has its own special spatial scale and orientation. Gabor feature extraction aims to extract the frequency and orientation information simultaneously. It is unnecessary to include all the filter parameters in frequency domain for image texture recognition. The scale of image texture feature will become bigger when the central frequencies become smaller. Gabor wavelets transform can perform scale and orientation information of signal excellently. 2D Gabor wavelets transform is a group of band-pass filters which can obtain local discriminant feature. Sub-image block ( and ) is transformed by 2D Gabor filter, and its Gabor feature description named as and can be defined in the equation below: (2)where is the input function with horizontal coordinate x () and vertical coordinate y (), and denote the even symmetry Gabor filter and the odd symmetry Gabor filter, respectively, is the centre frequency, is the orientation, s and d indicate the number of the scale and the orientation, respectively. According to the following experimental results analysis, the recognition can achieve the best performance when every Gabor wavelets filter contains four scales () and eight orientations (). 2.3 Fusion of the LG feature and are both complex numbers, which contain real part and imaginary part, and indicate the image texture from the aspects of different scales and orientations. The original image can be represented as 32 (; ) different scales and orientations Gabor sub-images and its high feature dimension could always lead to high redundancy. Fusion of the scale feature and orientation feature can completely describe richer Gabor texture feature. Fusion Gabor feature can not only decrease Gabor feature image's dimension, but also ensure information of the key decision not lost. First, different orientations of Gabor images with the same scale are fused by the fusion rules defined in the equations below: (3) (4)where and are the results of fusion with the same scale. In this paper, Gabor filters contain four scales () and eight orientations (). represents the (x,y)th image pixel point of the (i,j)th sub-image block. The decimal value for certain scalecan be represented by the equations below: (5) (6) Different orientations fusion images for can be represented by the equations below: (7) (8) where and are the Gabor different orientations fusionresults for with . Second, different scales of Gabor images with the same orientation are fused by the fusion rules defined in the equations below: (9) (10)The Gabor scale fusion images can be indicated as in the equation below: (11)Therefore, Gabor filters fusion results can be divided into two major parts: one is Gabor different orientations fusion results and and the other is Gabor different scale fusion results . To be more convenient for classification and recognition, the results need to be expanded along one row and combined to one vector. The Gabor different orientations fusion results can be combined as in the equation below: (12)where and are, respectively, represented by the equations below: (13) (14)where and are, respectively, represented by (15) and (16) with (15) (16)The Gabor different scales fusion results can be combined as in (17) and (18) (17)where is represented by (18) with (18)Finally, the texture feature description of is defined in the equation below: (19) 2.4 Creation of the adaptive weight map The information content of a random variable is supposed as entropy by (20) on the basis of Shannon theory, which probability is (20)The 2D entropy of is defined as in the equation below: (21)where m is the image grey degree, means probability of the ith grey level in the whole image and log() is the function of logarithm. The image entropy can represent richness of the image texture information, but it is unable to describe the image classification feature. The local image entropy in the local sub-image block can indicate a quantitative measure for the local image information content. The local entropy map (LEM) is calculated by a moving variable window and the certain image pixel local entropy is defined in the equation below: (22)where w indicates the variable window size; is the image entropy; and i and j indicate the pixel vertical coordinate and horizontal coordinate, respectively. is the sliding window which has the centre coordinate and defined in the equation below: (23)One original face and its LEMs are shown in Fig. 3. Fig. 3Open in figure viewerPowerPoint One original face and its LEMs (a) Original face, (b) w = 3, (c) w = 5, (d) w = 7, (e) w = 9 It is supposed that every pixel in the same local region has the same contribution to local image. The contribution map (CM) value of every local sub-image is defined in the equation below: (24)where q denotes the block number and is the pixel of the ith block in position . The sample face and CM with the different block numbers are shown in Fig. 4. Fig. 4Open in figure viewerPowerPoint Original face and CM with different block numbers (a) Original face, (b)q = 4, (c)q = 8, (d)q = 16, (e)q = 32 2.5 Histogram representation of weighted Gabor feature Histogram can always be used to describe the whole image texture effectively. However, some details of the local structure description will be lost when the histogram is calculated for the whole Gabor fusion image. Therefore, in this paper, the histograms are calculated for the local block Gabor fusion image, and then combined with each other to express the whole image feature. The histogram of local fusion Gabor image is defined in the equations below: (25) (26) can represent the feature of the local block image, but the importance of the local block image is not the same in the whole image. Thus, the weight of every local block image in the whole image is very significant. The histogram representation of weighted Gabor feature is defined by the equation below: (27) 2.6 Classifications Features obtained by each sub-pattern are used to train an RBF network. The size of weighted Gabor feature histogram is q×256, while q denotes the number of sub-image block. In this paper, the combined blocks mainly combine the 4, 8 and 16 blocks and the size of the feature vector is 28×256. If the rank of LDA is k and the person number of the face database is p, the size of input feature vector of the RBF is k×256 and the size of output of the RBF is p×1. 3 Experiments For evaluating performance of the proposed LWFG algorithm, simulated experiments are conducted on a number of publicly available face databases including ORL, Yale, CMUPIE, Yale B and FRET, which cover various conditions of partial occlusion, complex illumination, different expressions and poses. To guarantee the effectiveness of final results, every experiment was executed ten times for each database and final average results are reported. The proposed LWFG algorithm was evaluated by comparing with other typical algorithms including LBP [50], LGBP [62], LG [59], local principal components analysis (LPCA) [29], LTP [52] and some intermediate methods (local fusion gabor (LFG), local gabor histogram (LGH) and local fusion gabor histogram (LFGH)) whose abbreviations are shown as follows. It is worth noting that LGH is also presented by this paper which is similar to the proposed LWFG algorithm (see Fig. 5) just without fusion step. Fig. 5Open in figure viewerPowerPoint Abbreviations of some algorithms 3.1 Face databases 3.1.1 ORL face database ORL face database contains 400 grey-scale images in portable network graphics (PNG) format of 10 individuals and every individual has 40 images. All face images were normalised to 128×128 px2 and there are different facial expressions and poses in the database. Partial images of ORL are shown in Fig. 6. Fig. 6Open in figure viewerPowerPoint Partial images of ORL 3.1.2 Yale face database Yale database contains 15 individuals and every individual contains 11 images with different partial occlusion or expression conditions. Partial images of Yale are shown in Fig. 7. Fig. 7Open in figure viewerPowerPoint Partial images of Yale 3.1.3 CMUPIE face database CMUPIE database has 68 individuals and every individual contains 60 images with different poses, illuminations and expressions. Partial images of CMUPIE are shown in Fig. 8. Fig. 8Open in figure viewerPowerPoint Partial images of CMUPIE 3.1.4 Yale B face database Yale B database has 10 individuals and every individual contains 576 viewing condition images (9 different poses and 64 different illumination conditions). The extended Yale B dataset extends to 16,128 images of 28 individuals. Partial images of Yale B database are shown in Fig. 9. Fig. 9Open in figure viewerPowerPoint Partial images of Yale B 3.1.5 FRET face database FRET face database has 14,051 face images with different poses. There are five sets in the FRETdatabase such as Dup1, Dup2, Fa and more. Partial images of FRET are shownin Fig. 10. Fig. 10Open in figure viewerPowerPoint Partial images of FRET 3.2 Performance with different block numbers The number of sub-image blocks has a formative effect on performance of feature extraction algorithm. ORL, Yale, CMUPIE and FRET databases are adopted to verify the effectiveness of the proposed algorithm. The sample is divided into a series of blocks including 1, 2, 4, 6, 8, 16, 32 and combined blocks. The effect of combined blocks is tested to compare with 1, 2, 4, 6, 8, 16 and 32 blocks. In the experiment, the combined blocks model combines 4, 8 and 16 blocks and w is equal to 7 in (23). One face image is selected randomly from ten face images of database, which is used as the training sample. Meanwhile, the rest of individual images are used to be testing set. The recognition rates between the traditional algorithms and the proposed one with the change of block number are shown in Fig. 11. Fig. 11Open in figure viewerPowerPoint Recognition rate of algorithms with different block number (a) ORL database, (b) Yale database, (c) CMUPIE database, (d) FRET database It can be obtained that performance of the LWFG algorithm outperforms than other ones, i.e. LBP, LGBP, LG, LPCA, LTP, LFG and LGH with the change of block numbers. It indicates that the conditions of combined blocks are better than that of other block numbers and the recognition rate will decrease when the block numbers become bigger or smaller. When the block numbers increase, the texture description will fall into too meticulous local analysis and the holistic analysis will be weakened or even submerged. On the contrary, if the block size is too large, the texture description will fall into too holistic analysis and the local analysis will be weakened or even submerged. The combined blocks model can not only describe details of texture information, but also express whole structure information. 3.3 Performance with different scales and orientations Scales and orientations for Gabor filters can have different values. Suppose Gabor filters contain 2, 3, 4 and 6 scale types and 4, 6 and 8 orientation types. Gabor filters with different scales and orientations have different ability to express image's texture feature. Recognition rates comparison of algorithms related with Gabor are shown in Tables 1–3 with fusion of various scales and orientations. It is obvious that the identification efficiency will obtain the best when scale is set to four and orientation is set to eight. Results of the proposed LWFG algorithm are superior to other algorithms such as LGBP, LG, LFG, LGH and LFGH. Table 1. Recognition rate of algorithms with fusion of various scales and orientations on ORL databases Algorithm LGBP LG LFG LGH LFGH LWFG scale×orientation 2×4 78.25 76.25 75.25 78.50 78.75 80.75 2×6 79.75 78.50 80.25 79.00 79.50 81.25 3×4 80.75 81.00 79.00 80.25 79.75 81.50 3×6 81.00 80.25 81.75 81.00 77.50 80.50 4×4 81.50 79.75 81.00 79.75 81.50 82.75 4×8 82.25 81.50 82.75 81.25 81.50 84.75 6×4 82.00 80.00 81.50 80.75 81.00 82.25 6×8 81.75 81.25 80.50 79.50 80.25 81.00 Table 2. Recognition rates of algorithms with fusion of various scales and orientations on CMUPIE database Algorithm LGBP LG LFG LGH LFGH LWFG scale×orientation 2×4 71.49 71.04 72.51 73.19 69.12 76.24 2×6 73.64 74.55 71.38 74.32 69.57 76.81 3×4 72.74 71.83 72.74 77.26 74.55 78.39 3×6 72.96 72.62 76.58 74.32 76.24 77.60 4×4 77.04 76.70 75.45 75.79 77.26 79.41 4×8 78.39 77.83 77.49 79.07 80.32 81.56 6×4 73.30 77.71 74.77 73.42 75.68 79.64 6×8 75.57 75.11 75.90 74.32 72.62 78.05 Table 3. Recognition rates with fusion of various scales and orientations on FRET Algorithm LGBP LG LFG LGH LFGH LWFG scale×orientation 2×4 69.05 73.65 74.65 75.60 77.30 81.05 2×6 69.35 74.05 74.85 75.85 77.10 80.60 3×4 70.05 74.35 75.05 76.05 77.55 81.70 3×6 70.60 74.80 75.55 76.75 77.90 81.25 4×4 73.60 75.05 75.90 76.55 78.20 81.95 4×8 74.95 75.65 76.15 76.85 78.55 82.15 6×4 74.15 75.25 75.15 76.40 77.95 81.55 6×8 69.05 73.65 74.65 75.60 77.30 81.05 3.4 Experiments and analysis under partial occlusion Algorithm stability to occlusion is one key issue in face feature extraction. For the sake of evaluating algorithms' stability under partial occlusion, the proposed algorithm is systematically compared with LBP, LGBP, LG, LPCA, LTP, LFG, LGH and LFGH on the Yale with different block numbers. In the test, one occlusion face (with sunglasses) is randomly selected from every group, which is adopted as training sample. Meanwhile, rest of the images in every group within the same category are adopted as testing sets. Table 4 summarises recognition rates versus dimensions on Yale under partial occlusion based on different algorithms. It can be seen that recognition rates will achieve the best performance when the rank of LDA is eight and recognition rate will decrease when the rank of LDA becomes larger or smaller. At the same time, it is shown that the proposed algorithm consistently outperforms other algorithms. Table 4. Recognition rates versus dimensions on Yale under partial occlusion Algorithm Rank 2 4 6 8 9 10 11 LBP 60.00 62.42 64.24 66.06 63.64 61.82 60.00 LGBP 66.06 67.88 70.30 71.52 70.91 68.48 66.06 LG 61.21 64.85 65.45 67.88 64.24 63.03 61.21 LPCA 55.76 56.97 58.18 61.21 59.39 58.79 55.76 LTP 61.21 63.64 66.06 64.85 64.24 62.42 61.21 LFG 64.85 66.06 70.30 72.12 70.91 69.09 64.85 LGH 67.27 70.30 73.33 75.15 71.52 67.88 67.27 LFGH 70.91 74.55 76.97 78.18 76.36 73.33 70.91 LWFG 72.12 73.33 81.21 80.61 79.39 76.36 72.12 Bold values indicate the performance of proposed method. 3.5 Experiments and analysis under complex illumination The complex illumination is one of the critical challenges for face recognition. To test illumination robustness of the proposed algorithm, Yale B database is adopted. In the experiment, one different illumination face sample is selected from every set, which is adopted as training sample. Meanwhile, the rest of other images in every group are adopted as testing sample. Comparisons among all the algorithms on Yale B face database are tabulated in Table 5 under complex illumination condition. From this table, it can be seen that recognition rates will achieve best performance when the rank of LDA is eight and recognition rate will decrease when it becomes larger or smaller. At the same time, it is shown that LWFG algorithm consistently outperforms other algorithms under the complex illumination. Table 5. Recognition rates versus dimensions on Yale B under complex illumination Algorithm Rank 2 4 6 8 10 LBP 70.16 70.78 73.13 75.16 73.91 LGBP 67.50 72.97 74.69 76.88 75.94 LG 66.56 70.16 72.97 78.28 76.25 LPCA 62.66 64.06 66.09 71.72 68.28 LTP 66.72 74.22 70.78 77.50 75.31 LFG 68.13 71.09 75.78 78.75 78.28 LG