<figure><img src="image_1.png" /></figure>

HAL
open science

# Automatically Detected Feature Positions for LBP Based Face Recognition

Ladislav Lenc, Pavel Král

► To cite this version:

Ladislav Lenc, Pavel Král. Automatically Detected Feature Positions for LBP Based Face Recognition. 10th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Sep 2014, Rhodes, Greece. pp.246-255, 10.1007/978-3-662-44654-6_24. http-01391321

HAL Id: hal-01391321

https://inria.hal.science/hal-01391321

Submitted on 3 Nov 2016

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Automatically Detected Feature Positions for
LBP Based Face Recognition

Ladislav Lenc¹,², Pavel Král¹,²

¹Dept. of Computer Science and Engineering, University of West Bohemia, Plzeň,
Czech Republic

²NTIS, University of West Bohemia, Plzeň, Czech Republic
{llenc,pkral}@kiv.zcu.cz

**Abstract.** This paper presents a novel approach for automatic face recognition based on the Local Binary Patterns (LBP). One drawback of the current LBP based methods is that the feature positions are fixed and thus do not reflect the properties of the particular images. We propose to solve this issue by a method that automatically detects feature positions in the image. These key-points are determined using the Gabor wavelet transform and k-means clustering algorithm. The proposed method is evaluated on two corpora: AT&T Database of Faces and our Czech News Agency (ČTK) dataset containing uncontrolled face images. The recognition rate on the first dataset is 99.5% which represents 2.5% improvement compared to the original LBP method. The best recognition rate obtained on the ČTK corpus is 59.1% whereas the original LBP method reaches only 38.1%.

**Keywords:** Automatic Face Recognition, Czech News Agency, Gabor Filter, Local Binary Patterns

# 1 Introduction

Recognizing people from digitized images is already an old concept. A great progress was made during the last twenty-five years and the amount of proposed approaches is overwhelming. There are numerous applications of the face recognition such as an access control, tracking people or other systems where an authentication is necessary. There is also a great potential of face recognition in social networks and photo sharing applications.

The recognition of controlled images is considered as a well-resolved problem. However, the face recognition from the ordinary photographs is an open issue yet, because it requires very sophisticated algorithms.

Nowadays, the Local Binary Patterns (LBP) is a very popular algorithm for face representation and recognition. The strength of this algorithm lies in its high ability to capture important information in the images and in its low computational complexity. However, one weakness of the current LBP based methods is that the feature positions are fixed and thus do not respect the properties of the particular images. We would like to solve this issue by proposing
an approach to detect feature positions in images automatically. The key-points
candidates are determined using the Gabor Wavelet Transform (GWT). K-means
clustering algorithm is used subsequently to identify the final points. Another
improvement is to compare the features separately instead of concatenating them
into one feature as in the case of other existing LBP based methods. To the best
of our knowledge, no existing methods use the algorithms in this way. The results
of this work will be used by the Czech News Agency (ČTK<fnref n="1" />) to annotate people
in photographs during insertion into the photo-database<fnref n="2" />.

The rest of the paper is organized as follows. Section 2 summarizes important
face recognition methods with a particular focus on the local binary patterns.
The next section describes the LBP algorithm and the proposed face recognition
method. Section 4 evaluates the approach on the AT&T and ČTK corpora. In
the last section, we discuss the results and we propose some future research
directions.

## 2 Related Work

It is possible to divide the existing approaches into two main groups: holistic and feature based methods. The approaches from the first group represent the face image as a whole while methods belonging to the second group use for face representation a set of features. The successful holistic approaches are for example Eigenfaces [14], Fisherfaces [3] or Independent Component Analysis [2]. These methods achieve good results on controlled data. However, their performance decreases significantly when low quality real data are used.

This issue is partly solved by the feature based approaches. The most im-
portant representatives are detailed next. As a pioneer method we can consider
Elastic Bunch Graph Matching (EBGM) [15]. It is based on features created us-
ing Gabor wavelet transform. It is later used with many modifications as shown
for example in [6]. This method extracts the feature points automatically based
on the Gabor filter responses. Another type of features used for face representa-
tion is based on Scale Invariant Feature Transform (SIFT) [9]. The SIFT features
are invariant to rotation, scale and lighting conditions. Later a Speeded-Up Ro-
bust Features (SURF) is also used for face recognition as shown for instance
in [4]. It reaches comparable results as the SIFT, however it has significantly
lower computational demands.

In the last couple of years, also Local Binary Patterns (LBP) are successfully
used as features for face representation and recognition. The LBP operator was
first used for texture representation as presented in [10]. It is computed from the
neighbourhood of a pixel and uses the intensity of the central pixel as a threshold.
The pixels are marked either 0 or 1 if the value is lower or greater than this
threshold. The binary values are concatenated into one binary string and its
decimal value is then used as a descriptor of the pixel.

<footnote n="1">¹ http://www.ctk.eu</footnote>

<footnote n="2">² http://multimedia.ctk.cz/en/foto/</footnote>
The first application of LBP for face recognition is proposed by Ahonen et al. in [1]. The face is divided into rectangular regions. In each region a histogram of the LBP values is computed. All histograms are then concatenated into one vector which is used for the face representation. A histogram intersection method or Chi square distance are used for vector comparison. A weighted LBP modification is also proposed in this work. It gives more importance to the regions around the eyes and the central part of the face. The reported recognition rate on the FERET dataset [11] reaches 93% for the original method and 97% for the weighted LBP method.

A modification of the original LBP approach called Dynamic Threshold Local Binary Pattern (DTLBP) is proposed in [8]. It takes into consideration the mean value of the neighbouring pixels and also the maximum contrast between the neighbouring points. It is stated there that this variation is less sensitive to the noise than the original LBP method.

Another extension of the original method is Local Ternary Patterns (LTP) proposed in [13]. It uses three states to capture the differences between the center pixel and the neighbouring ones. Similarly to the DTLBP the LTP is less sensitive to the noise.

The so called Local Derivative Patterns (LDP) are proposed in [18]. The difference from the original LBP is that it uses features of higher order. It thus can represent more information than the original LBP.

An important idea which is proposed already by Ojala in [10] are so called uniform Local Binary Patterns. The pattern is called uniform if it contains at most two transitions from 0 to 1 or from 1 to 0. It was proved that approximately 90% of the patterns in facial images are uniform. The histogram can then be shortened from 256 intervals (bins) to 59, where the 59th bin is reserved for the non-uniform patterns.

An interesting method which uses uniform patterns is proposed in [17]. The authors state that the histogram bin containing non-uniform patterns dominates among other bins and gives thus too much importance to this bin. Therefore they propose to assign such patterns to the closest uniform pattern. Hamming distance is used for the face comparison.

Some methods also combine other preprocessing tools with the LBP. In [12] Gabor features and LBP are combined. The Gabor features as well as the LBP features are extracted and transformed using PCA. The features are then combined and used as face representation.

Another method called Local Gabor Binary Pattern Histogram (LGBPH) [16] also combines the Gabor wavelet transform and LBP. It first filters the image with a set of Gabor filters and obtains a set of magnitude images. Then the LBP operator is applied to each of the magnitude images.

Note that the common property of all above described LBP methods is that the images are divided into rectangular regions and histograms are computed in each region. All histograms from one image are concatenated and create the face representation.
# 3 Method Description

The proposed method is motivated by the assumption that the most representative features must be created in the “important” face points. This set of points where the features are created is calculated dynamically for each image by the Gabor wavelets. Therefore, the feature positions differ for each face image. We also propose to compare the features separately. In order to facilitate the reading of the paper, we shortly introduce next the Gabor wavelets and the LBP algorithm.

## 3.1 Gabor Wavelets

Gabor filter is a sinusoid modulated with a Gaussian. A basic form of a two dimensional Gabor filter is shown in Equation 1.

$$g(x, y; \lambda, \theta, \psi, \sigma, \gamma) = \exp\left(-\frac{\dot{x} + \gamma^2 \dot{y}^2}{2\sigma^2}\right) \cos\left(2\pi \frac{\dot{x}}{\lambda} + \psi\right) \quad (1)$$

where $\dot{x} = x \cos \theta + y \sin \theta$, $\dot{y} = -x \sin \theta + y \cos \theta$, $\lambda$ is the wavelength of the cosine factor, $\theta$ represents the orientation of the filter and $\psi$ is a phase offset, $\sigma$ and $\gamma$ are parameters of the Gaussian envelope, $\sigma$ is the standard deviation of the Gaussian and $\gamma$ defines the ellipticity (aspect ratio) of the function.

The Gabor wavelets are often used in image analysis because of their great ability to capture important information in images.

## 3.2 Local Binary Patterns

The original LBP operator uses the 3×3 neighbourhood of the central pixel. The algorithm assigns either 0 or 1 value to the 8 neighbouring pixels by Equation 2.

$$N = \begin{cases} 0 & \text{if } g_N < g_C \\ 1 & \text{if } g_N \ge g_C \end{cases} \quad (2)$$

where $N$ is the binary value assigned to the neighbouring pixel, $g_N$ denotes the gray-level value of the neighbouring pixel and $g_C$ is the gray-level value of the central pixel. The resulting values are then concatenated into an 8 bit binary number. Its decimal representation is used for further computation. This approach is illustrated in Figure 1.

The original LBP operator was further extended to use circular neighbourhoods of various sizes and also with different numbers of points. A bilinear interpolation is used to compute the values in the points that are not placed in the pixel centres. The LBP is then denoted as $LBP_{P,R}$ where $P$ is the number of points and $R$ is the radius of the neighbourhood.

Figure 2 depicts the original image and the LBP image after applying the original $LBP$, $LBP_{8,1}$ and $LBP_{8,2}$ operators.
<figure><img src="image_3.png" /><figcaption>Fig. 1. An example of the feature computing by the original LBP operator</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 2. An example of the original image and the LBP image after applying the original LBP, LBP<sub>8,1</sub> and LBP<sub>8,2</sub> operators</figcaption></figure>

3.3 Key-point Position Determination

The determination of the feature positions is probably the most important part
of the whole algorithm, particularly in the case of real world photographs where
the faces are extracted automatically. We decide to adopt and extend the idea
proposed by Kepenekci in [6]. In that work the fiducial points (positions where
the features are created) are extracted automatically by the Gabor wavelets.
A set of Gabor filters of different orientations and wavelengths is applied to
the original image and then the fiducial points are determined from the filter
responses.

The filtered images are scanned using a square sliding window W of the size
w × w. The window centre (x₀, y₀) is considered to be a fiducial point iff:

$$
R_j(x_0, y_0) = \max_{(x,y) \in W} R_j(x, y) \quad (3)
$$

$$
R_j(x_0, y_0) > \frac{1}{w_i * h_i} \sum_{x=1}^{w_i} \sum_{y=1}^{h_i} R_j(x, y) \quad (4)
$$

where *j* = 1, ..., *N* (*N* is the number of Gabor filters) and *w*ᵢ and *h*ᵢ are image width and height respectively.

**3.4 K-Means Clustering**

The number of points determined in the previous section is usually too high
(hundreds) and the points are often concentrated near important facial parts.
Moreover, a high number of the points increases significantly the computation
complexity. Therefore, we propose to use clustering to identify only the most important points. This idea is supported by the fact that usual LBP based methods use less than 100 points and achieve very good results. We chose the K-means algorithm to cluster the key-points.

Figure 3 shows points detected with different window sizes (15, 25, 35) and clustered points for the same window sizes and point counts (100, 75, 50).

<figure><img src="image_5.png" /><figcaption>Fig. 3. An example of the fiducial points determined with different window sizes (15, 25, 35) (left three ones) and the clustered points for the same window sizes and point numbers (100, 75, 50) (right three ones).</figcaption></figure>

## 3.5 Feature Construction

After determining the fiducial points, the features are constructed in these points. Each feature is described by its coordinates and the LBP histogram. The histogram is computed from a square window around the feature point. The resulting histogram is then normalized and can then be interpreted as a vector of the length 256.

In our application, we usually have several training images for each person. It is thus beneficial to use all available images to create the face model. We propose a *face specific* schema for model creation. The fiducial points determination algorithm and clustering are applied to each training image separately. Then, all obtained features are put together to create the face model of a given person.

## 3.6 Face Comparison

The comparison of two faces is based on the image features extracted according to the above described procedure. Each face is thus represented by a set of features that contain also feature point coordinates. Note that the number of the features in a face model may vary because the model is created from multiple images.

Contrary to the most of LBP based approaches we do not concatenate the feature histograms but compare the features separately. The Chi square statistic is used for computation of the distance of the features f and r:

$$ \chi^2(f, r) = \sum_i \frac{(f_i - r_i)^2}{f_i + r_i} \qquad (5) $$

The distance of two faces is defined as:
$$sim(F, R) = \sum_{f_i} \min_{r_j \in N(f_i)} (\chi^2(f_i, r_j)) \qquad (6)$$

where $N(f_i)$ is the neighbourhood of the feature $f_i$ defined by the *distanceThreshold* that specifies the maximum distance within that the features are compared. It means that for each feature of the face $F$ we find the closest feature within the neighbourhood $N(f_i)$ from the face $R$. The distance of the two faces is computed as a sum of these minimum distances.

The recognized face $\hat{F}$ is given by the following equation:

$$\hat{F} = \arg \min_{R} (sim(F, R)) \qquad (7)$$

# 4 Experimental Setup

## 4.1 Corpora

**AT&T Database of Faces** This database was formerly known as the ORL database. It was created at the AT&T Laboratories<fnref n="3" />. The pictures were captured between years 1992 and 1994 and contain the faces of 40 people. 10 pictures for each person are available. Each image contains one face with a black homogeneous background. They may vary due to three following factors: 1) time of acquisition; 2) head size and pose; 3) lighting conditions. The size of pictures is 92 × 112 pixels. A more detailed description of this database can be found in [5].

**Czech News Agency (ČTK) Database** This database is created from real-world photographs owned by the Czech News Agency. The dataset contains the images of 638 people. The database was created automatically from common photographs. The images have significant variations in pose, lighting conditions and also ageing of the objects.

The experiments described in this paper will be realized on the cleaned version of this corpus (see [7]), which contains up to 10 images for each of the 638 individuals. The testing part contains one image for each person whereas the rest of images are used for training. Note that only the testing part was checked manually.

## 4.2 Experiments on the AT&T Database of Faces

The first series of the experiments was carried out on the AT&T Database of Faces. Its purpose was to show the performance of our method on a controlled database where the face recognition is simple. Ten-fold cross-validation was used for evaluation.

Table 1 shows the recognition rates of the original LBP approach proposed by Ahonen [1] in comparison with our methods. We evaluated our method with

<footnote n="3">³ http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html</footnote>
three different types of the LBP operator: Original LBP operator, $LBP_{8,1}$ and $LBP_{8,2}$. The first two rows show recognition rates when 9 out of 10 images for each person are used for training and 1 for testing. The first row depicts the results with k-means clustering (50 clusters). The second row shows recognition rates without the clustering, it means we used directly all key-points detected by the Gabor wavelet based detection algorithm (see Section 3.3). The last row shows results for only one training example. It should demonstrate the differences between the methods when only one image for training is available. In this case, the clustering is also applied for the proposed approach.

<table><caption>**Table 1.** Recognition rates of the original LBP approach in comparison with our methods</caption><thead><tr><th rowspan="2">Method</th><th>Baseline</th><th colspan="3">Proposed approaches</th></tr><tr><th>LBP<sub>(Ahonen)</sub></th><th>LBP<sub>(original)</sub></th><th>LBP<sub>8,1</sub></th><th>LBP<sub>8,2</sub></th></tr><tr><th>Training images number</th><th colspan="4">Recognition rate (%)</th></tr></thead><tbody><tr><td>9 (k-means)</td><td>-</td><td>98.0</td><td>97.5</td><td>99.5</td></tr><tr><td>9 (no clustering)</td><td>97.0</td><td>97.8</td><td>97.8</td><td>99.8</td></tr><tr><td>1</td><td>60.8</td><td>66.8</td><td>66.4</td><td>68.8</td></tr></tbody></table>

The obtained recognition rates show that our method outperforms the original LBP method for all test configurations. Using the $LBP_{8,2}$ operator gives the best results. The difference is apparent especially in the case when only one training image is used. If we compare the results with and without clustering, the recognition rates are comparable. However, omitting the clustering causes significantly increased number of features and longer computation time. Therefore the clustering seems to be a good choice for feature number reduction.

## 4.3 Experiments on the ČTK Database

The remaining experiments were performed on the ČTK database. As a baseline we chose (similarly as in the previous section) the LBP based face recognition algorithm proposed by Ahonen et al. [1]. The recognition rate obtained by this baseline algorithm is only 38.1%.

In this experiment (see Table 2), we compare this baseline with the recognition results and computation times of our proposed approaches with and without the clustering. The original LBP operator is used.

This table shows clearly that both proposed configurations outperform significantly the original approach. Moreover, the clustering decreases significantly the computation time with almost no negative impact on the classification accuracy.

The last experiment (see Table 3) shows recognition rates of our algorithm with varying number of clusters for three LBP operator types.
<table><caption>**Table 2.** Recognition rates and computation times of our proposed approaches with and without the clustering, the original LBP operator is used, tested on the Intel Core i5-2300, 2.80GHz, 16GB RAM.</caption><thead><tr><th></th><th>Clustering</th><th>No clustering</th></tr></thead><tbody><tr><th>Recognition rate (%)</th><td>56.4</td><td>55.3</td></tr><tr><th>Computation time (s)</th><td>6,905</td><td>17,476</td></tr></tbody></table>

<table><caption>**Table 3.** Recognition rates for different numbers of clusters. Three types of LBP operators are evaluated.</caption><thead><tr><th>LBP type</th><th>LBP<sub>(original)</sub></th><th>LBP<sub>8,1</sub></th><th>LBP<sub>8,2</sub></th></tr><tr><th>Cluster count</th><th colspan="3">Recognition rate (%)</th></tr></thead><tbody><tr><td>50</td><td>45.6</td><td>45.1</td><td>56.4</td></tr><tr><td>75</td><td>47.8</td><td>46.9</td><td>58.6</td></tr><tr><td>100</td><td>47.5</td><td>46.1</td><td>59.1</td></tr></tbody></table>

This table shows clearly that the $LBP_{8,2}$ operator outperforms significantly the other two ones. The highest recognition rate is obtained with 100 clusters.

## 5 Conclusions and Perspectives

We proposed and evaluated a new approach for face recognition based on the LBP features. The main contribution is using automatically detected feature positions instead of fixed positions used in usual LBP based approaches. We use Gabor wavelets for key-point detection. Then, these points are clustered by the k-means algorithm. The next contribution is proposal of a new algorithm for face comparison based on the Chi square statistic. It compares individual features within specified distance. The proposed method is evaluated on the AT&T database of faces and on the ČTK dataset. The recognition rate for the AT&T database is 99.5%. For the more challenging ČTK dataset we obtained recognition rate of 59.1%. We significantly outperformed the baseline approach in all cases.

One possible perspective is applying weighting to the automatically detected key-points. Another possible improvement may be using only uniform patterns or some modification of LBP algorithm such as LTP or LDP.

## Acknowledgements

This work has been partly supported by the UWB grant SGS-2013-029 and by the project "NTIS - New Technologies for Information Society", European Centre of Excellence, CZ.1.05/1.1.00/02.0090. We also would like to thank ČTK for support and for providing the photographic data.
References

1. Ahonen, T., Hadid, A., Pietikäinen, M.: Face recognition with local binary patterns. In: Computer vision-eccv 2004, pp. 469–481. Springer (2004)

2. Bartlett, M.S., Movellan, J.R., Sejnowski, T.J.: Face recognition by independent component analysis. Neural Networks, IEEE Transactions on 13(6), 1450–1464 (2002)

3. Belhumeur, P.N., Hespanha, J.P., Kriegman, D.: Eigenfaces vs. fisherfaces: Recognition using class specific linear projection. Pattern Analysis and Machine Intelligence, IEEE Transactions on 19(7), 711–720 (1997)

4. Dreuw, P., Steingrube, P., Hanselmann, H., Ney, H., Aachen, G.: Surf-face: Face recognition under viewpoint consistency constraints. In: BMVC. pp. 1–11 (2009)

5. Jain, A.K., Li, S.Z.: Handbook of face recognition. Springer (2005)

6. Kepenekci, B.: Face recognition using gabor wavelet transform. Ph.D. thesis, Cite-seer (2001)

7. Lenc, L., Král, P.: Face recognition under real-world conditions. In: International Conference on Agents and Artificial Intelligence, Barcelona, Spain (2013)

8. Li, W., Fu, P., Zhou, L.: Face recognition method based on dynamic threshold local binary pattern. In: Proceedings of the 4th International Conference on Internet Multimedia Computing and Service. pp. 20–24. ACM (2012)

9. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 2 (2004)

10. Ojala, T., Pietikäinen, M., Harwood, D.: A comparative study of texture measures with classification based on featured distributions. Pattern recognition 29(1), 51–59 (1996)

11. Phillips, P.J., Wechsler, H., Huang, J., Rauss, P.: The FERET database and evaluation procedure for face recognition algorithms. Image and Vision Computing 16(5), 295–306 (1998)

12. Tan, X., Triggs, B.: Fusing gabor and lbp feature sets for kernel-based face recognition. In: Analysis and Modeling of Faces and Gestures, pp. 235–249. Springer (2007)

13. Tan, X., Triggs, B.: Enhanced local texture feature sets for face recognition under difficult lighting conditions. Image Processing, IEEE Transactions on 19(6), 1635–1650 (2010)

14. Turk, M.A., Pentland, A.P.: Face recognition using eigenfaces. In: Computer Vision and Pattern Recognition, 1991. Proceedings CVPR’91., IEEE Computer Society Conference on. pp. 586–591. IEEE (1991)

15. Wiskott, L., Fellous, J.M., Kuiger, N., Von Der Malsburg, C.: Face recognition by elastic bunch graph matching. Pattern Analysis and Machine Intelligence, IEEE Transactions on 19(7), 775–779 (1997)

16. Xie, Z.: Single sample face recognition based on dct and local gabor binary pattern histogram. In: Intelligent Computing Theories, pp. 435–442. Springer (2013)

17. Yang, H., Wang, Y.: A lbp-based face recognition method with hamming distance constraint. In: Image and Graphics, 2007. ICIG 2007. Fourth International Conference on. pp. 645–649. IEEE (2007)

18. Zhang, B., Gao, Y., Zhao, S., Liu, J.: Local derivative pattern versus local binary pattern: face recognition with high-order local pattern descriptor. Image Processing, IEEE Transactions on 19(2), 533–544 (2010)