Genetic‐based feature fusion in face recognition using arithmetic coded local binary patterns

IET Image ProcessingVolume 14, Issue 15 p. 3742-3750 Research ArticleFree Access Genetic-based feature fusion in face recognition using arithmetic coded local binary patterns Saeed Najafi Khanbebin, Saeed Najafi Khanbebin Department of Electrical and Electronics, Lorestan University, Khorramabad, IranSearch for more papers by this authorVahid Mehrdad, Corresponding Author Vahid Mehrdad Mehrdad.v@lu.ac.ir Department of Electrical and Electronics, Lorestan University, Khorramabad, IranSearch for more papers by this author Saeed Najafi Khanbebin, Saeed Najafi Khanbebin Department of Electrical and Electronics, Lorestan University, Khorramabad, IranSearch for more papers by this authorVahid Mehrdad, Corresponding Author Vahid Mehrdad Mehrdad.v@lu.ac.ir Department of Electrical and Electronics, Lorestan University, Khorramabad, IranSearch for more papers by this author First published: 18 February 2021 https://doi.org/10.1049/iet-ipr.2020.0394AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Local binary patterns (LBPs) are one of the attempts for gathering local features with face recognition algorithms. Although the application of LBP's in many recognition contents is too apparent, these methods have limited accuracy because of their threshold value. One problem is earning one value for two different regions with a diverse pixel neighbourhood, which causes mistakes in feature vector and decreases the discriminative power. In this study, the authors proposed a modified LBP that covers the LBP's disadvantages. The proposed approach is arithmetic coded LBP (ACLBP) that uses arithmetic coding process during LBP calculation instead of applying original thresholds. The proposed policy addresses the problem of returning one similar LBP value for two different patches. Moreover, the proposed method modifies LBP by using a different threshold for calculating the pixels differences. Using this algorithm, the authors conducted a genetic-based feature fusion method by combining LBP and histogram of oriented gradients and ACLBP. The proposed approach could work better on LFW dataset, and the ORL dataset and Yale face dataset that shows the improving role of ACLBP in comparison with the earlier version of LBP. 1 Introduction A vital part of recognition researches over the last decades has been related to face recognition problems. Face recognition is one of the fields of image analysis and computer vision that is based on computer–human interaction [[1]]. In computer vision topics, face recognition is one of the essential contents for the researchers, with extensions to perceptual, behavioural, and social principles [[2]]. It is applicable in a variety of domain such as in automated teller machine, healthcare system, driving license system, railway reservation system, surveillance operation, and passport authentication [[3]]. Last decades a wide range of face recognition algorithms have been employed such as using eigenfaces in which the eigenvectors of the set of face in the face spaces used [[4]], recognition using class specific linear projection that takes advantage of projected images while discounting the face regions with large deviation [[5]], using independent component analysis, where the pixels are treated as outcomes and the images as random variables graph embedding and dimensionality reduction in which the available projection directions and data distribution assumptions made it outperform the traditional algorithms [[6]], performing discriminative multi-manifold analysis with the discriminative features learned from the images [[7]], face description with LBPs where an enhanced feature vector is derived from the local binary pattern (LBP) distribution [[8]], scale-invariant feature transform (SIFT) features for face authentication in which the application of SIFT is investigated in the face authentication context [[9]] using multi-scale local image structures, and speeded up robust features (SURF) face which takes the advantage of SURF features [[10]]. Although lots of methods have been applied in face recognition to address this content's problems, there is no reason to ignore today's researchers’ interests in this content because there is a considerable amount of unsolved problems in uncontrolled conditions. In many real-world images with full scenes, it is challenging to recognise holistic images because faces are in wild conditions and might be occluded [[11]]. Recently, researchers have used these steps for 2D face recognition: face detection was the first step, and after that secondly, face alignment, feature extraction located in the third step and finally feature matching was in fourth step [[3]]. Some methods are based on seeking an efficient feature to a better performance in face recognition [[12], [13]]. Feature extraction methods mainly consist of two categories: holistic features in subspaces and local features [[14]]. There are a lot of face representation methods that have been studied in many papers [[15]]. LBP is the name of one of the essential face representation methods because of its high performance in classification based on a texture that is the result of its robustness against global intensity variations [[15]]. For the suitable returns gained from the first version of LBP [[16]], researchers proposed lots of methods based on LBP's. A multi-scale block LBP was presented by Liao et al. [[17]]. In this type, instead of the pixel itself, mean values of pixels’ sub-region are under the LBP operation. The next LBP-based method that applies the feature extraction process in four directions (0_, 45_, 90_, and 135_) was local derivative pattern [[18]]. There are a considerable number of approaches based on local features. These methods have some advantages against holistic approaches. Still, original LBP-based approaches suffer from calculation faults. The reason is the nature of that kind of LBP which is the base of these methods. Since the threshold level in these methods is not suitable for every scene, its discriminative power has real difficulties over all kinds of images. These faults are mainly in the feature extraction step of the face recognition process. In the feature extraction step in this paper, we proposed a new LBP calculation that can address the threshold problem of LBP-based methods that has reached suitable accuracy over face datasets. Arithmetic coded LBP (ACLBP) is the way that can solve this problem and is used in our present study. Arithmetic coding is one of the compression methods that is used in our process by the way that codes each sequence of intensities to a number between one and zero that is a unique one. Using this coding method instead of the simple original LBP calculation process with more innovations results in sufficient performances that solved the calculation faults of LBP-based approaches. Besides the neural network-based algorithms, the hand-crafted features are used for conducting the main method. This paper proposes a new procedure to make a progressive step in this field. Furthermore, the presented process shows that this content still has a lot of potential for researches, especially in the face recognition task. This study proposes a feature extraction method that works with local features in face images. This type of features returns a summarised form of an image that decrease the computational costs in the recognition process. Neural networks (such as CNN etc.) are different types of feature extraction and recognition tools, which require lots of images to work well. In other words, the neural network feature extraction approaches work with the original image and the features are learnt from all the individual pixels that lead to a heavy computational cost. The leading structure of this paper is as follows. First, the fundamentals of LBPs and related studies are presented. In the second section, the principal methodology of our approaches is shown in the way that the description of algorithms that are used in the proposed method are presented in details to find out the fundamental theories of presented algorithms. In continuous, the results of the experiments are presented, and finally, the conclusion is the rest of the present paper. 2 Related works Face recognition tasks have been one of the most interesting subjects among the researchers. A brief review of previous works is presented here to have the right view about the former researches in this content. Before delivering texture and LBP-based approaches, some face recognition papers are provided. 2D-discrete cosine transform (DCT) distance based face recognition using a reduced number of coefficients [[19]] is a face recognition research that performs its application in three steps. The attributes are extracted in the feature extraction step. The best group of features are chosen in the training step to apply the identification task. In the last step, unknown samples are compared with the extracted features to complete the classification step. This research used ORL dataset for its experiment. Face recognition using DWT and eigenvectors [[20]] is another approach in this field. This research, by the use of discrete wavelet transform (DWT), each image is divided into four parts. Then the principle component analysis (PCA) is used to lower the dimension of data and prevent from confusing calculations. After all, the feature vector is conducted by the DWT and PCA. Yale face dataset is used for the experiment part of this research. One critical part of computer vision and specifically in face recognition processes is texture analysis. This importance has attracted the researchers to this topic and its applications. LBPs with low computational cost and discriminative performance in texture analysis play an important role in this context. The power and successful use of the original LBP led to convincing researchers to use it as the main tool for their challenges in texture analysis and face recognition tasks. This interesting performance is the reason that many LBP-based methods were proposed by many researchers last years, and many LBP variants have been modified to take advantage of the LBP operator in research topics. In this section, a review of former approaches in the literature is presented. A modified LBP is symmetric (CS_LBP), rotated LBP, uniform LBP, and an appliance of LBP on convolutional neural networks (LBPNet) [[21]]. There is also a successful combination of LBP and Gabor features in facial expression applications [[22], [23]]. One type of LBP that in the time of calculating the final descriptor considers the LBP signs differences is rotated LBP that can be used in emotion recognition applications [[23]]. An enhancement of LBP with higher dimensions and more extended feature vector that is more robust against variations than LBP is symmetric (CS_LBP) that successful experiments of this type on the facial feature extraction have been applied on flatter images [[22], [23]]. The LBPs can be enhanced by the use of SIFT descriptors [[24], [25]]. An appliance of LBP on Gabor magnitude responses is local Gabor BP [[26]], and development by local Gabor XOR patterns [[22]] is also proposed in this content. One approach with higher resistance against rotation and noise is the dominant LBP, in which dominant patterns from the other patterns are selected to form the feature vector [[27]]. Taking advantage of a combination of the Weber local descriptor and LBP led to propose the Weber local binary patterns [[28]]. The LBP of the hue component of HIS colour space is a utilised form of the LBP for colour images and brought the more distinctive ability to it. This approach is based on thresholding colour pixels in the local neighbourhood in two categories [[29]]. Local colour vector binary patterns method have been proposed for the face recognition uses, which have colour angular and colour norm parts [[30]]. Table 1 shows a list form for some of the other LBP-based approaches with their applications. Table 1. Some of the LBP variants and their applications Method Application LBP [[31]] texture classification LTP [[32]] face recognition LDP [[33]] texture classification LMEBP [[34]] image retrieval CRLBP-S-MxC [[35]] texture classification DDLBP [[36]] face recognition DCP [[37]] face recognition LCCMSP [[38]] texture classification QBP [[39]] target tracking LDTP [[40]] texture classification RALBGC [[41]] texture classification 3 Theoretical background 3.1 Binary patterns LBP is one of the most critical information. Since visual cognition of majority of all living organisms is based on LBP, it plays a crucial component role for computer vision system [[42]] which is obvious in all multispectral remotely sensed data to microscopic images [[43]]. The LBP is an approach in texture analysis that consists of the integration of statistical and structural models which is robust against monotonic greyscale changes and has a simple computation [[43]]. This method uses local texture features from the image in a small equally spaced neighbourhood with P pixels on the R radius. The difference between neighbourhood pixels’ intensity and the central pixel is calculated. For each pixel P, if the difference value is positive, the pixel is assigned to 1 and 0 otherwise that provides a binary sequence. Then the total LBP label for this central pixel equals to the summing binary sequence values weighted with power 2 [[44]] LB P P , R = ∑ P = 0 P − 1 s g p − g c 2 P (1) s ( x ) = 0 , O . W 1 , x ≥ 0 (2) where g P is the neighbourhood pixel P’s greyscale value and the g c is the greyscale value of the central pixel [[44]]. The s(x) refers to the core equation of LBP, where 1's and 0's are generated in mask according to the other values of the former equation. 3.2 Arithmetic coding Arithmetic coding is a powerful method for data compression that is an entropy coding method [[45]]. This method has some differences from the others that there is information about the exact relation between the actual values of bits and coded symbol [[46]]. The pattern that is used in the arithmetic method is representing a message by a number between 0 and 1. Fig. 1 clearly shows that how the arithmetic coding process works. When a source is joined in, this number reduces depending on the coded source symbols in the way that more likely symbols are gathered with less range reduction [[48]]. There is no need to build entire codebook in this type of coding; the code for the tag corresponding to a sequence can be obtained. Fig. 1Open in figure viewerPowerPoint Restricting the interval containing the tag for the input sequence {α1, α2, α3,.} [[47]] By providing a unique identifier, this method enables us to distinguish a sequence of symbols from the others. This unique identifier is the unit interval [0, 1), and it can be suitable because of the infinite numbers in the range [[47]]. As the fundamental of the arithmetic coding is presented, the encoding process consists of a step. This procedure creates a sequence of nested intervals in the below form: Φ s = ⌊ α k , β k , k = 0 , 1 , … , N (3) where S is the source data sequence, ∝ k , β k represent real numbers such 0 ≤ α k ≤ α k + 1 , β k + 1 ≤ β k ≤ 1, while N shows the number of symbols in a sequence which is going to be coded [[46]]. For more explanation in a step-by-step form, first entire the half-open interval 0 , 1 is considered for the complete sequence. Based on the probabilities of the source symbols, the interval is divided into four parts. The first subinterval is related to the first symbol of the sequence. This subinterval is narrowed and in the next step stretched to all the entire interval [0,1), and the endpoints are places as labels and the narrowed values are held. Just like before, the restricted interval is divided into four parts which are done based on the original source symbols probabilities. Then the process goes to the next symbol. These contracting and stretching the intervals are the steps that repeat for each symbol until the last symbol and last subinterval. Each number in the last interval can be a representation of the entire sequence [[49]]. Besides the detailed explanation of the arithmetic coding process that is prepared in the former lines, to gain a more detailed observation about the step of this algorithm that is used in our proposed method, a programming shape of this algorithm can help. So, here is the pseudocode of the arithmetic encoding algorithm [[50]] (Fig. 2). Fig. 2Open in figure viewerPowerPoint Pseudocode of the arithmetic encoding algorithm 3.3 Histogram of oriented gradients Histogram of oriented gradients (HOG) calculates the gradient magnitude and direction that represents itself as an excellent descriptor that has been successful in object detection and recognition. The HOG computation steps are organised as follows. First, the gradient amplitude of image is calculated both in the horizontal and vertical directions with a 1_D mask G x ( x , y ) = I ( x + 1 , y ) − I ( x − 1 , y ) (4) G y ( x , y ) = I ( x , y + 1 ) − I ( x , y − 1 ) (5) In these equations, I(x, y) is the pixel value of the point (x, y) and G x and G y denote horizontal and vertical gradient amplitude, respectively. Then the gradient of colour images must be calculated. In this way first, the gradient of colour image channels (R, G, and B) should be calculated, and after that, one of the largest magnitude values is selected as the pixel gradient G ( x , y ) = G x ( x , y ) 2 + G y ( x , y ) 2 (6) where G(x, y) is the amplitude of the pixel(x, y) θ x , y = tan − 1 G x x , y G y x , y (7) where θ is the pixel gradient direction, in the third step, each block is divided into cells. The Gaussian weights are added by these blocks to adjust the gradient amplitude, each bin range is k π 9 , k + 1 π 9 , k = 0 , 1 , . . . . . , 8 . The fourth step is the step of concatenating the histogram of each block [[51]]. By this process, the primary feature extraction step of our approach is conducted. So, it is time to use the genetic algorithm (GA) as an optimisation part. 3.4 Genetic algorithm Darwinian natural selection and genetics are the base of the adaptive optimisation search of GA method that is presented as an alternative for conventional methods. The population is a set of candidate solution that the GA uses it. The optimal solution after iterations and computations is obtained by GA based on the Darwinian principle of ‘survival the fittest’. Although GA can operate exploration searches, it can perform large search spaces as well, and of course, the chance of getting local optimal solutions is less than the others. A fitness function is used to evaluate the solution quality. Leading operators of GA are crossover and mutation functions that randomly impact the fitness value. Reproduction by assessing the fitness value is to select chromosomes. The higher probability of being chosen for the GA operation represents the fitter chromosome [[52]]. While the feature extraction part is calculating the features from images, GA tells us how to fuse these features to achieve the best available result. 4 Methodology As is presented in Fig. 3, the main structure of our proposed method contains three main steps namely face detection, feature extraction, and classification. This section includes the fundamentals of the main idea of this paper. It describes how does the method calculates the texture feature to be gathered with earlier plans to improve classification accuracy. The definition of the algorithms that is used in this method is the other part of the second section. The main idea of our study is based on using arithmetic coding while calculating LBP so this part of the paper is described step by step to figure out clearly how does the method operate the calculations over the pixel values. Fig. 3Open in figure viewerPowerPoint 9×9 sliding window with selected pixels Although the discriminative power and the successes that LBP method has affected a huge number of researches, some faults of the LBP performances over images have attracted many researchers to present a new way to develop the original LBP operator. One of these faults is because of the threshold is used in the original LBP that leads to calculation faults, for example, it is going to calculate a similar binary sequence for two different matrixes that result in mismatches in feature vector construction and then in the recognition process. This study provides a new LBP calculation that has solved calculation faults of original LBP to achieve better performance over recognition problems 4.1 Arithmetic coded LBP The principal methodology of our feature extraction is to use arithmetic coding process to find texture features. It is a kind of LBP which is used in many recognition problems. However, there is a difference between LBPs and our approach that is using arithmetic coding during extracting local patterns. In the procedure of developing LBP, firstly a 9×9 sliding window is conducted. It is similar to the image that is presented in Fig. 3. Then for each pixel and its surroundings, the difference between a threshold (central pixel) and its neighbourhood intensities in the sliding window from selected pixels are calculated, which returns positive and negative numbers. As a novel way in LBP calculation for each block, we used some threshold values instead of the central pixel. These values are neighbourhood pixels, the average, median, and maximum of each block pixels’ intensities. In ACLBP, we store positive and negative values separately in two matrixes. Fig. 4 shows the positive and negative matrixes that are ACLBP_P and ACLBP_N, respectively. This figure shows a more readable format and easier understanding of the calculations which result in the negative and positive matrix. The 3×3 masks are used instead of the 9×9 that is the base of our method and presented in Fig. 4. This replacement is for preventing from trapping into the mixed-up calculations. Then these matrixes (in Fig. 4) are fed to arithmetic coding process that returns a unique number between one and zero for each matrix. Also, a probability of numbers in the sequence of neighbourhood pixels is needed to calculate the arithmetic code of them. The probability factor shows the probability distribution of a value in a sequence. This factor indicates the existence of each number in a sequence which will be converted into arithmetic coded value. Fig. 4Open in figure viewerPowerPoint Positive and negative matrixes that are the result of calculating the difference between the central pixel and its neighbourhood pixels Fig. 5Open in figure viewerPowerPoint Main procedure of the main method To address this problem, (4) is used as a probability factor. This parameter depends on the ‘times’ coefficient that presents how many of this number is repeated in this sequence. By this process, two matrixes are made while each one consists of the results calculated by the arithmetic coding that is substituted with the central pixel of the 9×9 mask. These matrixes are our feature vectors. This approach enables us to claim that the calculation fault of LBP which is because of the wrong binary sequence of pixel neighbourhoods is addressed by our method because two matrixes for each block prevent from this type of calculation faults. It is because the arithmetic coding returns a unique number between one and zero for each block and is not dependent on the threshold and after that to the code length p = ( unique symbol × times ) / sum ( seq ) (8) Here the seq coefficient presents the sequence of neighbourhood pixels and the unique symbol denote the intensity value that is going to be used in the arithmetic coding process. Then these vectors are concatenated to build up a unit feature vector for each image. This process is illustrated in Fig. 6. As is clear from Fig. 6, the image should be converted to greyscale, and the ACLBP window is earned from the gridded greyscale image. Fig. 6Open in figure viewerPowerPoint ACLBP procedure Working with the digest form of images is what the feature extraction approaches bring us to make the classification tasks done with a lower level of computational costs. Thus, proposing a feature extraction method that works well on face datasets can be a progressive step in face recognition problems. This property is the reason that although the increasing uses of neural network and learning features, the hand-crafted feature extraction method still attract the researcher's interests to this content. A multi-stage feature extraction is an interesting tool used by researches that makes classifications more successful. This approach covers the limitation of one method by the use of advantages on the other processes in other stages. For example, besides the benefits like decorrelation and energy compaction in DCT algorithm, this method is not robust against illumination directions and suffers from lack of spatially invariant magnitude of its coefficients. Also in DWT approach its ability in localizing signals simultaneously in time and scale is the reason of its usefulness [[19]]. This characteristics lead to shift variance in the downsampling process and does not have the directional selectivity. This means the extra stage is needed to cover the limitation of these approaches. Since the LBP-based algorithms and the proposed method in this paper (ACLBP) extract texture features from images, using HOG with its geometric characteristics and gathering these texture and geometric features in a combination form results in a more robust final feature vector. Choosing the best way of combining the components is a major problem that presents how much each stage affects the final feature vector. In our proposed method, this problem is addressed by the GA, which solves the optimisation problem that leads to the most optimised path that the features from stages can be combined. This optimised combination increases the discriminative power of the proposed method. 4.2 Genetic-based feature fusion Improving the classification accuracy is done by finding the best feature combinations. The GA cost function consists of weighted feature vectors that are fed to SVM classifier to find the best solution that is the optimised feature vector. Cost = W 1 × ACLBP _ P , W 2 × ACLBP _ N , W 3 × LBP , W 4 × HOG (9) where ACLBP_P and ACLBP_N are the positive and negative matrix of the proposed method, respectively, the LBP and HOG refer to the feature vector of these methods and W's are the weights of each element in this equation that earns by the GA and shows how they affect the final feature vector. The feature vector is now ready to be used in the classification step. Support vector machines (SVMs) is the choice of our paper for this step. 4.3 Support vector machines One of the most powerful algorithms in a supervised manner for classification is SVMs which initially was not designed for multiclass classification at first. A linear function w T x + b can express the decision boundary of SVMs, where w ∈ R d is the weight vector, and b ∈R is the shift of the separating hyperplane [[43]]. After conducting a feature vector of ACLBP implemented on gallery images, we applied this algorithm on probe images and, the feature vector of probe images is ready to be fed in classifiers to test the classifier that was trained by gallery images feature vectors. In this study, the multiclass SVM classifier is used for the classification step. 5 Experimental results This study aims to develop the face recognition accuracy using an ACLBP feature and then a genetic-based feature fusion procedure. For this purpose, the first input image is fed to the face alignment step that crops the face region, and it is the input image of the feature extraction layer. In this step at first, a new LBP is presented, called ACLBP, which is extracted from the face image. Then the HOG and original LBP features of the image are extracted too to be selected by the GA method. In this step, the GA uses optimised weights to each separate feature vector of all features (ACLBP and HOG and LBP) and then an optimised combination of these feature vectors that result in the efficient distinctive power with the most accuracy is returned. The classification layer of this procedure is operated by the multiclass SVM method. The dataset that is used in this paper is the labelled face in the wild (LFW) [[53]] and the Olivetti Research Laboratory (ORL) [[54]] dataset. The variations in viewpoint and background and on the other side, different facial expression and details are the factors that can affect the performance of our proposed method. The LFW, ORL, and Yale face datasets consist of the images with these variations and the performance on these datasets prove the efficiency of our approach. In this study, a batch of LFW dataset is selected (four classes) to show how the model performs on variat