<figure><img src="image_1.png" /></figure>

HAL
open science

# Enhanced LBP-Based Face Recognition System Using a Heuristic Approach for Searching Weight Set

Nhat-Quan Huynh Nguyen, Thai Hoang Le

► To cite this version:

Nhat-Quan Huynh Nguyen, Thai Hoang Le. Enhanced LBP-Based Face Recognition System Using a Heuristic Approach for Searching Weight Set. 13th IFIP International Conference on Computer Information Systems and Industrial Management (CISIM), Nov 2014, Ho Chi Minh City, Vietnam. pp.547-558, 10.1007/978-3-662-45237-0_50 . hal-01405642

HAL Id: hal-01405642

https://inria.hal.science/hal-01405642

Submitted on 30 Nov 2016

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Enhanced LBP-based face recognition system
using a heuristic approach for searching weight
set

Nhat-Quan Huynh Nguyen and Thai Hoang Le

Department of Computer Science, Ho Chi Minh City University of Science
1012337@student.hcmus.edu.vn, lhthai@fit.hcmus.edu.vn

**Abstract.** Local Binary Patterns is one of the most effective approaches for pattern recognition in general and face recognition in particular. There have been many studies on improving this method such as changing the input values or using another kind of histogram. Although weight set is also an important key leading to the success of this method, it does not seem to get much attention. A majority of LBP-based approaches are still using the weight set of Ahonen et al.'s study, one of the first researches applying LBP to face recognition. In this study, we introduce a powerful algorithm named Heuristic Weight Search, which finds a suitable weight set for not only LBP-based approaches but also other methods using weight set to improve performance. Experiments on the FERET database prove an ability of HWS thanks to their higher accuracy than original methods.

**Keywords:** Face recognition; Local binary patterns; Heuristic search; FERET; Weight set.

# 1 Introduction

Face recognition is a highly active area of researchers thanks to its real-world applications. Up to now, a large number of face recognition methods have been developed. Local Binary Patterns (LBP) is widely used for face recognition based on its potential for speed and accuracy. LBP can be applied for not only face recognition, but also many applications such as: texture classification, facial expression recognition and gender recognition. Since Ahonen et al. [2] introduced LBP as a powerful method for describing faces and reported impressive results in experiments on the FERET database in 2004, many LBP-based descriptors have been proposed. In 2012, a research from Ngoc-Son Vu et al. [9] introduced the Patterns of Oriented Edge Magnitudes (POEM) descriptor which uses LBP operator as a main part of this powerful descriptor.

According to Ahonen et al.'s research [1], a weight set is computed through a simple procedure. Although this weight set has been used for many researches to improve performance without any changes, it seems to be too general to fix the characteristic of each data set. That makes the problem how to create an
<figure><img src="image_3.png" /><figcaption>Fig. 1: Architecture of face recognition system in use.</figcaption></figure>

efficient weight set for a respective condition of images such as facial expression or illumination changes.

In this paper, we propose a Heuristic Weight Search (HWS) algorithm to solve the problem of generating weight set based on the properties of each image set. Applying this algorithm on any face recognition system using weight set helps to improve performance at all. A sample system integrating POEM descriptor into HWS algorithm is also introduced.

Our study is organized as follows: in section 2 and 3, LBP and POEM are introduced respectively; section 4 represents our algorithm to generate weight set of image sets; the sample system combining POEM descriptor and HWS algorithm is illustrated in section 5; the next section presents our experiments on FERET database; conclusion and references are in the rest of this paper.

## 2 Local Binary Patterns

Being introduced by Ojala et al. [5] in 1996, LBP operator is a powerful tool for texture description. This operator labels the pixels of an image and thresholds each neighbourhood of 3 × 3 pixels by using the central pixel value. The gray value of each pixel $g_p$ in the neighborhood is compared to the gray value $g_c$ of the central pixel. If $g_p$ is greater than $g_c$, then it is assigned 1, and 0 if not. The LBP label for the central pixel of image is obtained as [1]:

$$LBP = \sum_{p=0}^{7} S(g_p - g_c)2^p, \quad \text{where } S(x) = \begin{cases} 1, & x \ge 0 \\ 0, & x < 0 \end{cases} \qquad (1)$$

The basic operator was extended to use neighbourhood of different sizes to capture dominant features at different scales. Notation LBP(P, R) denotes a neighborhood of P equally spaced sampling points on a circle of radius of R. For example, the original LBP operator can be denoted LBP(8, 1).

According to [6], there is a small subset of 2^p patterns, called uniform patterns, accounted for the majority of the texture of images, over 90% of all patterns for LBP(8, 1) and about 70% for LBP(16, 2). These patterns contain at most two bitwise transitions from 0 to 1 or vice versa for a circular binary pattern. For instance, ‘00000000’ (0 transition), ‘01111100’ (2 transitions), and ‘10001111’ (2 transitions) patterns are uniform while ‘11001110’ pattern is not
<figure><img src="image_4.png" /><figcaption>Fig. 2: LBP(8, 1) operator.</figcaption></figure>

uniform due to its four bit transitions. The uniform patterns can be used to find the pixels that belong some texture primitives such as spot, flat area, edge, and corner. It can be denoted as $LBP_{(P,R)}^{u2}$, which was mentioned in [2].

<figure><img src="image_5.png" /><figcaption>Fig. 3: LBP feature extracting procedure in two case of applying weight set.</figcaption></figure>

In the face recognition procedure of Ahonen et al. [2], a facial image is divided into m small same sized non-overlapping regions called patches from which LBP histograms are extracted and concatenated into a single vector, which enhanced the facial representative ability. A patch histogram is defined as [1]:

$$H_i = \sum_{x,y} I\{LBP(x, y) = i\}, \quad i = 0, 1, \dots, n-1 \qquad (2)$$

in which n is the number of different labels produced by the LBP operator and

$$I(A) = \begin{cases} 1, & A \text{ is true} \\ 0, & A \text{ is false.} \end{cases} \qquad (3)$$
In using classifier like k-Nearest Neighbour to decide label for input image, distance computing formula must be used in order to calculate distance between images which is difference between feature vectors of these images. It was also announced by psychophysics studies that some regions on face contain more useful information than others in the process of identifying a person. That means each region on face has its own contribution which can be represented as a number called weight. Difference between applying specific and default weight set is presented in Fig 3. In this research, we suggest using weighted Chi square statistics, which has been used to build a dissimilarity measure of face with the weight set W.

$$ \chi^2_W(S, M) = \sum_{i,j} w_j \frac{(S_{i,j} - m_{i,j})^2}{S_{i,j} + M_{i,j}} \quad (4) $$

where S and M are two LBP histogram, $w_j$ is the weight for jth region.

<figure><img src="image_6.png" /><figcaption>Fig. 4: Ahonen et al.'s weight set for the weighted $\chi^2$ dissimilarity measure. Black square indicate weight 0.0, dark gray 1.0, light gray 2.0, and white 4.0.</figcaption></figure>

## 3 Patterns of Oriented Edge Magnitudes

In their research, Ngoc-Son Vu et al. [9] introduce a LBP-based structure on oriented edge magnitudes called Patterns of Oriented Edge Magnitudes (POEM). This descriptor uses the gradient magnitudes instead of intensity value in the computing using LBP operator.

POEM feature is extracted through the process shown at Fig. 5. Firstly, the oriented edge magnitude image (oriented EMI) generated from an original input image is divided into m uni-oriented EMIs ($uEMIs$ ($uEMI^{\theta_i}$)) through gradient orientations of pixels, over $0 - \pi$ (unsigned representation) or $0 - 2\pi$ (signed representation). Then, each pixel on $uEMIs$ is then replaced by the sum of values of its $w \times w$ neighbour area called cell. The resulting images are called accumulated EMIs (AEMIs). On the next stage, POEM images are created by applying $LBP^{u2}$ operator on AEMIs. Finally, the output POEM feature called POEM-HS (POEM histogram sequence), which is the concatenation of LBP histograms of all POEM images, is generated:
<figure><img src="image_7.png" /><figcaption>Fig. 5: POEM descriptor extracting procedure.</figcaption></figure>

$$POEM = [POEM^{\theta_1}, ..., POEM^{\theta_m}], \quad POEM^{\theta_i} = LBPH^{u2}(AEMI^{\theta_i}) \quad (5)$$

In their study, Ngoc-Son Vu et al. [9] made many experiments on the FERET database and got the optimal POEM parameters for face recognition as: an unsigned representation with three direction bins ($m = 3$) built on 7 × 7 pixels cell using $LBP_{(8,5)}^{u2}$. These experiments were made on face images of 110 × 100 pixels. The similar optimal parameters were obtained when testing with images of around 90 × 90 pixels.

# 4 Heuristic Weight Search (HWS)

Based on proposed approach presented on a submitted paper [8], we made some changes and show our work here. In this section, we introduce an algorithm called Heuristic Weight Search (HWS) which uses heuristic search to find an optimal weight set. Firstly, we show how to prepare data for HWS. Then, fitness function evaluating goodness of a weight set, a way to prioritize patches and a complete HWS algorithm is presented in order.

## 4.1 Organizing data

A certain database is divided into three image sets for using through weight search and evaluating accuracy process:
- Gallery set ($S_G$) contains images of everyone in database, in which each person has only one image. Both training and testing stage need to use $S_G$ to decide labels of images.

- Training set ($S_{Tr}$) includes images captured in different condition such as expression or illumination from those in Gallery set. It does not require containing images from all people in database. Training set is used in weight search process and does not involve with testing stage.

- Testing set ($S_{Ts}$) also has images captured in the similar condition to those in Training set, which is only used in computing accuracy of system.

In process of HWS, only $S_G$ and $S_{Tr}$ are used to generate an efficient weight set. $S_{Ts}$ is needed in evaluating performance to prove a power of proposed algorithm. Detailed content is presented in section 6.

## 4.2 Weight set fitness function (WsFitness)

There are many face recognition systems using weight set to enhance their performance. It cannot be denied that weight set plays an important role on these systems. Therefore, anyone can also easily wonder how to know a suitable weight set for their data, or which is the best weight set from a several ones. To solve this problem, we suppose each weight set has its own fitness value for a certain image set. A basic idea of fitness value of a weight set is a recognition rate of system in case of using that weight set. The higher value is, the better weight set is. We call a function calculating fitness value of a weight set WsFitness.

$$ WsFitness(W) = \frac{\left| \left\{ \begin{array}{l} I_{Tr} | I_{Tr} \in S_{Tr}, \text{label}(I_{Tr}) = \text{label}(I), \\ \qquad I = \underset{I_G \in S_G}{\operatorname{argmin}} (\chi_W^2(I_{Tr}, I_G)) \end{array} \right\} \right|}{|S_{Tr}|} \quad (6) $$

where label($X$) is identification of image $X$ and $\underset{x \in S}{\operatorname{argmin}}(f(x))$ shows the object of $x$ producing the smallest value in computing with function $f(x)$.

## 4.3 Different contributions of patches

Each region on a face has its own role in process of face recognition. Calculating contributions of all patches is needed before processing proposed algorithm. In this subsection, we define a single entry weights matrix denoted $sEM(i)$ whose values are zero except an element in position of $i$. If we apply this matrix to face recognition process, only nonzero patch will affect recognition rate. Therefore, we call a fitness value of $sEM(i)$ a contribution of $i$th patch. We also name the matrix containing these values a contribution matrix $CM$.
<figure><img src="image_8.png" /><figcaption>Fig. 6: Contribution matrix in an experiment.</figcaption></figure>

## 4.4 Heuristic Weight Search in detail

After calculating a contributions of all patches, heuristic search is applied to find an optimal weight set whose elements are scaled on [0, 1] range. First of all, we initialize a new weight set $W_n$ with 0.0 for all weights except that the most important patch, which gains the highest contribution, is assigned a value of 1.0. From now on, all changes in weight values are processed in $W_n$. Then, weight values of other patches are calculated in descending order of the contributions of patches, which use a similar procedure presented as follows: (1) weight of a certain patch decreases steadily from 1.0 to 0.0 by small static changing step of 0.05 (read 6.2 for more details); (2) the value maximizing $W_n$ is an optimal weight of that patch.

### Heuristic Weight Search (HWS) algorithm

1. Compute a descending ordered contribution list $CM$.

2. Initialize new weight set $W_n = \{0, \}\$; set weight for the most important patch $W_n[index(CM[0])] = 1$.

3. Loop for each remaining value in $CM$, find the best weight value for each patch, which is in range of [0, 1] and maximizes $W_n$.

4. The optimal weight set $W_n$ is ready for application.

In above presentation of HWS, $index(X)$ returns a respective position of patch having fitness value of $X$.

## 5 Combination

In this part, we introduce a face recognition system applying POEM descriptor to HWS in order to show a boosting ability of the proposed algorithm.
<figure><img src="image_9.png" /><figcaption>Fig. 7: Architecture of HWS-POEM evaluating system.</figcaption></figure>

Fig. 7 presents the system integrating POEM descriptor into HWS algorithm, which is called HWS-POEM system. As can be seen from that diagram, our proposed system follows the procedure of normal face recognition system, except a few changes. That means people can easily apply HWS-POEM combination to their certain system to improve performance.

## 5.1 Pre-processing

<figure><img src="image_10.png" /><figcaption>Normalized image</figcaption></figure>

1. Integer to float conversion - Converts 256 gray levels into floating point equivalents.

2. Geometric normalization - Lines up human chosen eye coordinates.

3. Masking - Crops the image using an elliptical mask and image borders such that only the face from forehead to chin and cheek to cheek is visible.

4. Histogram equalization - Equalizes the histogram of the unmasked part of the image.

5. Pixel normalization - Scale the pixel value to have a mean of zero and a standard deviation of none.

<caption>Fig. 8: Image normalization using pre-processing tool of CSU.</caption>

Condition of input images make a huge impact on recognition result. If we apply a powerful descriptor on bad quality images, we might get a worse result than applying a normal descriptor on good pre-processed images. In this work, we use pre-processing tool of CSU Face Identification Evaluation System [3] to ensure that condition of images are better. Fig. 8 shows five steps in converting a PGM FERET image to a normalized image.

Certainly, there are many better methods of image pre-processing, for instance, that of Ngoc-Son Vu et al. [9]. Although applying these methods improves performance, we only use CSU tools in all of our experiments in order to evaluate approaches on FERET database.
## 5.2 Discussion

This system inherits abilities from both POEM descriptor and HWS algorithm, so it has their characteristic.

POEM has many properties which are mentioned in [9]; two most important ones are: (1) POEM is an oriented feature so that it has the ability to capture image information with different levels of orientation accuracy, (2) using gradient magnitudes instead of intensity values makes POEM more robust to lighting variations than original methods of LBP.

HWS algorithm brings us a number of benefits. Firstly, HWS algorithm can be applied to any face recognition systems, provided weight set can be used in that system. In other words, if features of facial parts can be calculated separately using a face descriptor, we can use our algorithm there. Secondly, After finding weight for each patch, we could get the best weight in range of [0, 1]. With the first assigned value 1, a new calculated value could be better or equal to the weight of 1. Even in the worst case, we might still get the weight set whose elements are all 1, which means a new recognition rate of system applying weight set from HWS algorithm could not be less than that of original system. Finally, HWS algorithm just generate an efficient weight set without integrating into applications. Therefore, speeds of these applications are not be affected.

# 6 Experiments

## 6.1 Database

In our experiments, we use facial images from FERET database [7]. This database contains 14051 face images which are divided into many subsets based on their condition. According to FERET Tests September 1996, some frontal image subsets were classified as below, all changes of non-Gallery (we can name it Probe) images are in comparison with these ones in Gallery set.

<table><caption>Table 1: FERET frontal image subsets.</caption><thead><tr><td>Image set</td><td>Quantity</td><td>Description</td><td>Image list</td></tr></thead><tbody><tr><td>Gallery</td><td>1196</td><td>Single images of all subjects</td><td>gallery.names</td></tr><tr><td>FAFB</td><td>1195</td><td>Images having changes in face expression</td><td>probe_fafb_*.names</td></tr><tr><td>FAFC</td><td>194</td><td>Images captured under different illumination</td><td>probe_fafc_*.names</td></tr><tr><td>DUP1</td><td>722</td><td>Images taken between 0 to 1031 days after</td><td>probe_dup_1_*.names</td></tr><tr><td>DUP2</td><td>234</td><td>Subset of DUP1, contains images captured at least 18 months after</td><td>probe_dup_2_*.names</td></tr></tbody></table>

For experiments, Gallery set of FERET database is used as Gallery set $S_G$ of HWS algorithm. Training set $S_{Tr}$ and testing set $S_{Ts}$ is generated by cross-validation k-fold method with k = 2 for each FERET's Probe set. The final result of a certain Probe set is an average of experiments on that image set.
## 6.2 The parameter of step value in search

<figure><img src="image_11.png" /><figcaption>Fig. 9: Fitness changes in comparison between steps of 0.05 (left) and 0.01 (right) in weight search of the 2nd (top) and 4th (bottom) patches (based on an importance of patches). Vertical axis is fitness value; horizontal one is weight of a certain moment.</figcaption></figure>

First of all, we made some experiments to compare effect on fitness (section 4.2) of weight change value in HWS process. Firstly, it can be easily to aware that a large step makes weight search faster. Secondly, as can be seen from Fig. 9, in the early stages of HWS process, both fitness graphs of 0.05 and 0.01 share the same pattern despite a difference of step value. It means effect of 0.05 step value on fitness is similar to 0.01 one. In addition, Fig. 10 shows the fitness values of a new weight set after searching weight for each patch. Although spending much time on calculating, the experiment using step of 0.01 reach the final fitness slower than that of another one. In another experiment in which a step value of 0.1 is used, it also reaches the final fitness slower than that of 0.05. This problem can be explained as 0.1 is such a large weight change that it pass through optimal weight values of patches while 0.01 is so small value that over fit training data.

Finally, using a small step of weight change like 0.01 is too waste and brings no more benefits than a medium value like 0.05 while a weight change of 0.1 leads to some missing optimal values. Therefore, 0.05 is a suitable weight step for HWS algorithm.

## 6.3 Result

We made experiments using HWS algorithm to find an optimal weight set for some LBP-based methods. Experiments of the same descriptor use the same parameters. In the table of results below, we call Ahonen et al.'s weight set by Ahonen and weight set found by Heuristic Weight Search algorithm by HWS.
<figure><img src="image_12.png" /><figcaption>Fig. 10: Fitness of new weight set after completely searching weight for each patch in comparison between steps of 0.05 (left) and 0.01 (right). Vertical axis is fitness value; horizontal axis is the completion of new weight set (maximum value is 100).</figcaption></figure>

<table><caption>Table 2: Boosting performance ability of HWS algorithm in comparison with original approaches.</caption><thead><tr><th>Method</th><th>FAFB</th><th>FAFC</th><th>DUP1</th><th>DUP2</th></tr></thead><tbody><tr><td>LBP [2]</td><td>93.0</td><td>51.0</td><td>61.0</td><td>50.0</td></tr><tr><td>LBP + Ahonen [2]</td><td>97.0</td><td>79.0</td><td>66.0</td><td>64.0</td></tr><tr><td><strong>LBP + HWS</strong></td><td><strong>98.33</strong></td><td><strong>84.07</strong></td><td><strong>67.88</strong></td><td><strong>71.90</strong></td></tr><tr><td>MLBP</td><td>92.38</td><td>28.28</td><td>61.09</td><td>49.59</td></tr><tr><td>MLBP + Ahonen</td><td>97.24</td><td>58.76</td><td>65.65</td><td>61.11</td></tr><tr><td><strong>MLBP + HWS</strong></td><td><strong>98.24</strong></td><td><strong>85.07</strong></td><td><strong>66.49</strong></td><td><strong>71.79</strong></td></tr><tr><td>POEM</td><td>96.82</td><td>77.85</td><td>72.73</td><td>71.08</td></tr><tr><td>POEM + Ahonen</td><td>98.22</td><td>86.88</td><td>71.83</td><td>70.21</td></tr><tr><td><strong>POEM + HWS</strong></td><td><strong>99.08</strong></td><td><strong>90.73</strong></td><td><strong>75.07</strong></td><td><strong>77.35</strong></td></tr><tr><td>LGBP</td><td>98.16</td><td>96.41</td><td>78.95</td><td>76.50</td></tr><tr><td>LGBP + Ahonen</td><td>98.91</td><td><strong>98.45</strong></td><td>80.47</td><td>80.77</td></tr><tr><td><strong>LGBP + HWS</strong></td><td><strong>99.16</strong></td><td>97.42</td><td><strong>82.41</strong></td><td><strong>83.33</strong></td></tr></tbody></table>

There are some differences (worse or better) between results of our experiments and those reported in original papers (POEM [9]; Multi-scale LBP - MLBP [4]; Gabor+LBP - LGBP [10]) because of different pre-processing methods (mentioned in 5.1) and matching algorithms (presented in section 2). However, in our experiments, each method gains a higher performance than its original when applying weight set found by HWS algorithm, which proves the effectiveness of proposed algorithm. As can be seen from the results of LGBP group, the performances of all experiments when applying our algorithm to find weight set are higher than those of LGBP approach and LGBP using weight set of Ahonen et al. [2], except an experiment on FAFC subset where the recognition rate of LGBP using this weight set is higher than found weight set from HWS algorithm. This can be explained that LGBP is so suitable for illumination changes on images in FAFC subset that it does not boost significantly a final performance when applying HWS algorithm for this experiment. In this case, a hand labeled weight set like that of Ahonen et al. [2] is good enough to use.
# 7 Conclusion

This paper introduces a new algorithm named Heuristic Weight Search (HWS) to find an optimal weight set for any face recognition system applying it. The combination system of HWS algorithm and POEM descriptor also shows an ease of integrating proposed algorithm to a certain face recognition system. Experiments on FERET database helps to analysing the parameter of weight change and suggests a suitable step value. The performances of systems integrating proposed algorithm proves its ability in comparison with other approaches.

# Acknowledgement

This research is funded by Vietnam National University of Ho Chi Minh City (VNU-HCMC) under the project "Features descriptor under variation condition for real-time face recognition application", 2014. We are also supported by research funding from Advanced Program in Computer Science, University of Science, Vietnam National University - Ho Chi Minh City.

# References

1. Ahonen, T., Hadid, A., Pietikäinen, M.: Face recognition with local binary patterns. In: Computer vision-eccv 2004, pp. 469–481. Springer (2004)

2. Ahonen, T., Hadid, A., Pietikainen, M.: Face description with local binary patterns: Application to face recognition. Pattern Analysis and Machine Intelligence, IEEE Transactions on 28(12), 2037–2041 (2006)

3. Bolme, D.S., Beveridge, J.R., Teixeira, M., Draper, B.A.: The csu face identification evaluation system: its purpose, features, and structure. In: Computer Vision Systems, pp. 304–313. Springer (2003)

4. Chan, C.H., Kittler, J., Messer, K.: Multi-scale local binary pattern histograms for face recognition. Springer (2007)

5. Ojala, T., Pietikäinen, M., Harwood, D.: A comparative study of texture measures with classification based on featured distributions. Pattern recognition 29(1), 51–59 (1996)

6. Ojala, T., Pietikainen, M., Maenpaa, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. Pattern Analysis and Machine Intelligence, IEEE Transactions on 24(7), 971–987 (2002)

7. Phillips, P.J., Moon, H., Rizvi, S.A., Rauss, P.J.: The feret evaluation methodology for face-recognition algorithms. Pattern Analysis and Machine Intelligence, IEEE Transactions on 22(10), 1090–1104 (2000)

8. Thai, L.H., Khoa, T.Q.D., Vu, N.D.: On approaching heuristic weight mask to improve face recognition. International Journal of Pattern Recognition and Artificial Intelligence (IJPRAI) (submitted)

9. Vu, N.S., Dee, H.M., Caplier, A.: Face recognition using the poem descriptor. Pattern Recognition 45(7), 2478–2488 (2012)

10. Zhang, W., Shan, S., Gao, W., Chen, X., Zhang, H.: Local gabor binary pattern histogram sequence (lgbphs): A novel non-statistical model for face representation and recognition. In: Computer Vision, 2005. ICCV 2005. Tenth IEEE International Conference on. vol. 1, pp. 786–791. IEEE (2005)