# Unconstrained Face Identification Based on 3D Face Frontalization and Support Vector Guided Dictionary Learning

Authors: Zhi Zhang; Xin Xu; Jiuzhen Liang; Bingyu Sun
Journal: Mathematical Problems in Engineering
 (2020)

---

## Abstract

Face identification aims at putting a label on an unknown face with respect to some training set. Unconstrained face identification is a challenging problem because of the possible variations in face pose, illumination, occlusion, and facial expression. This paper presents an unconstrained face identification method based on face frontalization and learning-based data representation. Firstly, the frontal views of unconstrained face images are automatically generated by using a single, unchanged 3D face model. Then, we crop the face relevant regions of the frontal views to segment faces from the backgrounds. At last, to enhance the discriminative capability of the coding vectors, a support vector-guided dictionary learning (SVGDL) model is applied to adaptively assign different weights to different pairs of coding vectors. The performance of the proposed method FSVGDL (frontalization-based support vector guided dictionary learning) is evaluated on the Labeled Faces in the wild (LFW) database. After decision fusion, the identification accuracy yields 97.17% when using 7 images per individual for training and 3 images per individual for testing with 158 classes in total.


---

## Body

## 1. Introduction

As one of the biometric technologies, face recognition has developed rapidly in recent decades. Compared with other biometric technologies such as fingerprint and iris recognition, face recognition has clear advantages of being natural and noncontact. Face recognition includes face verification and face identification, and the objective of face verification is to predict whether an image pair is from the same person or not, and the objective of face identification is to put a label on an unknown face with respect to some training set.During the last few decades, a lot of research studies have been carried out on face recognition and considerable progress can be seen. Despite this, unconstrained face recognition is a very challenging problem because of the possible variations in face pose, expression and illumination, etc. Traditional methods for frontal face images cannot handle with these variations; therefore, recently tremendous efforts have been put on developing robust face descriptors, face frontalization methods, and learning-based face recognition methods.Popular face descriptors can be divided into two categories: global representing method such as Principle Component Analysis (PCA) [1], Linear discriminate Analysis (LDA) [2], hand-crafted local features such as Local Binary Pattern (LBP) [3], and Gabor wavelets [4]. However, these face descriptors are sensitive not only to pose changes but also to occlusion and expression variations.Face frontalization is the process of synthesising a new, frontal view of a face image with varying face pose. Recent reports have suggested that an effective face frontalization method may greatly improve the performances of face recognition systems. Given a detected face and its landmarks, Fontaine et al. [5] proposed a computationally efficient 2D mesh-warp method to align it to a reference face. Morphable-based methods [6–8] attempt to estimate 3D shape and texture of faces from single image by fitting a statistical, morphable 3D face model to images, while these morphable-based methods require near-frontal of unoccluded faces. Kemelmacher-Shlizerman and Basri [9] proposed a shape from the shading method which uses harmonic representations of lighting to recover the 3D shape of the given face, and this method can produce outstanding facial details. Despite this, the method is sensitive to occlusions, and it requires segmenting the faces from the backgrounds carefully. An effective face frontalization method was proposed in [10] by using a single, unchanged 3D face model to generate the frontal view of a given face. This method is not only computationally efficient but also outstanding in preserving facial details.Sparse representation has seen a rapid development both in theory and algorithms during the past few years. The Sparse Representation-based Classification (SRC) scheme that Wright et al. [11] proposed first applied the ideas of sparse coding to face recognition which have made great success. While in [12], Zhang et al. claimed that it is the collaborative representation but not the ℓ1-norm sparsity that makes SRC powerful for face recognition. Although these methods have shown their ability to face recognition, directly using the whole training samples as the dictionary atoms may make them not effective enough to represent the query images. Dictionary Learning (DL) is a learning-based method that obtains a dictionary by learning from the training samples. Dictionary learning methods can be divided into two categories: supervised and unsupervised methods. One of the representatives of unsupervised DL is the K-SVD algorithm [13]; Jiang et al. [14] added label consistency constraint to the original K-SVD algorithm to enforce the discrimination of coefficient vectors. However, the dictionary that K-SVD learned is an overcomplete dictionary, which means it could only faithfully represent the training samples but is not suitable for classification. Therefore, lots of supervised dictionary methods have been proposed. Sprechmann and Sapiro [15] proposed a method that uses the sparse representation coefficients to learn a subdictionary for each class and successfully applied it to face recognition and clustering. Based on Fisher discrimination criterion, Yang et al. proposed a Fisher discrimination dictionary learning (FDDL) algorithm in [16]. This class-specific dictionary learning method learns a subdictionary for 60 each class, and fisher discrimination criterion is also performed on the coefficient vectors. However, this method needs sufficient training samples for each class to achieve good performance, and so it is not suitable for small training sample problems. Recently, a parametrization method (Support Vector-Guided Dictionary Learning, SVGDL) which can adaptively determine the weight of each coding vector pair was proposed in [17]. This method only imposes constrains on the coefficients and has achieved good performance on face identification.In this paper, we focus on the challenging problem of unconstrained face identification. Our contributions can be summarized as follows:(1)
We propose a frontalization-based support vector guided dictionary learning (FSVGDL) method to cope with unconstrained face identification. On one hand, face frontalization can reduce the impact of pose variations by automatic synthesis, a new, frontal view of a face image. The process of frontalization also helps segment the faces from their backgrounds since it synthesizes the frontal views in the center of the image. The removal of backgrounds can simplify unconstrained face identification. To address other variations in unconstrained images, we learn a support vector-guided dictionary which can adaptively assign different weights to different pairs of coding vectors for face representation. Therefore, the coding vectors have strong discriminative capability to represent face images well.(2)
Decision fusion generally leads to performance gain; therefore, we perform fusion on several decisions. We use voting to do the fusion, despite its simplicity, and the experiment results show the effectiveness of decision fusion.The rest of the paper is organized as follows. In Section2, we provide a brief view of the related works. The detailed process of the proposed FSVGDL (frontalization-based support vector-guided dictionary learning) method is introduced in Section 3. The experimental results on LFW (Labeled Faces in the wild) database are presented in Section 4. Finally, Section 5 concludes this paper.


## 2. Related Works

### 2.1. Whitened Principal Component Analysis

Whitened Principal Component Analysis is a method that performs whitening after Principal Component Analysis. Principal Component Analysis (PCA) is a method that we commonly used to reduce the noise in feature vectors; whitening can make the feature vectors more discriminative. PCA computes thed eigenvectors with the largest eigenvalues of the covariance matrix. The covariance matrix of PCA is defined by(1)ΣPCA=∑i=1nti−mti−mT,where tl˙ denotes an image feature vector, m is the mean of the image feature vectors, ∧=λ1,…,λd denotes the d eigenvalues, and ∨=v1,…,vd denotes the corresponding d eigenvectors of ΣPCA. The whitening process is denoted by(2)t^=diagλ1−1/2,…,λd−1/2∨Tt,where t∧ denotes the feature vector after whitening PCA. We can see from equation (2) that the features are weighted by the inverse of the eigenvalues. Through this method, we can penalize the eigenvectors with large eigenvalues and therefore make the input feature vectors less redundant.


### 2.2. General Discriminative Dictionary Learning

Suppose there areK classes of subjects T=T1,T2,…,TK and Ti∈ℜm×niTi=t^1,t^2,…,t^ni is a m-dimensional training vector from class i. Denote N=∑i=1Kni the total number of training samples in the dataset, and ni is the subset number of training samples from class i. The learned dictionary is denoted as D=D1,D2,…,DK, and V=V1,V2,…,VN are the coefficient vectors of T over D. The general discriminative dictionary learning (DDL) model can be described as follows:(3)D,V=argminD,VℜT,D,V+λ1VpP+λ2LV,where λ1 and λ2 are the trade-off parameters, ℜT,D,V is the reconstruction term, p denotes ℓp-norm, and L(V) denotes the discrimination term.Note that, to enhance the discriminative capability of a dictionary learning-based method, we can either introduce a discrimination term on the coefficient vectors (e.g., SVGDL [17] method) or perform structure constraints on the learned dictionary (e.g., FDDL [16] method).


## 2.1. Whitened Principal Component Analysis

Whitened Principal Component Analysis is a method that performs whitening after Principal Component Analysis. Principal Component Analysis (PCA) is a method that we commonly used to reduce the noise in feature vectors; whitening can make the feature vectors more discriminative. PCA computes thed eigenvectors with the largest eigenvalues of the covariance matrix. The covariance matrix of PCA is defined by(1)ΣPCA=∑i=1nti−mti−mT,where tl˙ denotes an image feature vector, m is the mean of the image feature vectors, ∧=λ1,…,λd denotes the d eigenvalues, and ∨=v1,…,vd denotes the corresponding d eigenvectors of ΣPCA. The whitening process is denoted by(2)t^=diagλ1−1/2,…,λd−1/2∨Tt,where t∧ denotes the feature vector after whitening PCA. We can see from equation (2) that the features are weighted by the inverse of the eigenvalues. Through this method, we can penalize the eigenvectors with large eigenvalues and therefore make the input feature vectors less redundant.


## 2.2. General Discriminative Dictionary Learning

Suppose there areK classes of subjects T=T1,T2,…,TK and Ti∈ℜm×niTi=t^1,t^2,…,t^ni is a m-dimensional training vector from class i. Denote N=∑i=1Kni the total number of training samples in the dataset, and ni is the subset number of training samples from class i. The learned dictionary is denoted as D=D1,D2,…,DK, and V=V1,V2,…,VN are the coefficient vectors of T over D. The general discriminative dictionary learning (DDL) model can be described as follows:(3)D,V=argminD,VℜT,D,V+λ1VpP+λ2LV,where λ1 and λ2 are the trade-off parameters, ℜT,D,V is the reconstruction term, p denotes ℓp-norm, and L(V) denotes the discrimination term.Note that, to enhance the discriminative capability of a dictionary learning-based method, we can either introduce a discrimination term on the coefficient vectors (e.g., SVGDL [17] method) or perform structure constraints on the learned dictionary (e.g., FDDL [16] method).


## 3. Frontalization-Based Support Vector-Guided Dictionary Learning

In this section, we will introduce the proposed frontalization-based support vector-guided dictionary learning method in detail. The framework of our method is shown in Figure1.Figure 1
Illustration of the proposed method.


### 3.1. Face Frontalization Method

Given a query image IQ and its landmarks, we can synthesis its frontal view. The following are the detailed steps for face frontalization. By specifying a projection matrix [18] CM=AMRMTM, where matrix AM denotes the intrinsic parameters and RMTM denotes the extrinsic parameters consisting of a rotation matrix RM and a translation vector TM, we can produce a synthetic view of the textured 3D face model. In this paper, we select an extrinsic matrix CM to produce the frontal view IR of the 3D face model, as shown in Figure 2. This frontal view IR will be used as our reference image as well as the reference coordinate system next.Figure 2
The synthetic reference face of the 3D face model.During the process of producing reference view, for each pixelp′ in the reference view, we store its corresponding 3D point coordinates P=X,Y,ZT located on the 3D face model. The correspondence between P and p′ can be obtained by(4)p′∼CMP.We denoteP=X,Y,ZT as the facial feature points of the query image IQ, and p′=xi′,yi′T are the same facial feature points of the reference view IR. From equation (4), we have the 3D coordinate point Pi=Xi,Yi,ZiT corresponding to the facial feature pi=x,iyiT of the query image. This provides the correspondences piT,PiT=xi,yi,Xi,Yi,Zi between the query image and the 3D face model; thus, we can estimate the projection matrix MQ=AQRQTQ, which is approximately the one used to capture the query image.With the projection matrix MQ, by using the expression,(5)p∼MQP.We have the correspondences between each pixel of the query image and each 3D coordinate of the 3D face model. Therefore, we have correspondences between each pixel of the query image and each pixel of the reference face. By using bilinear interpolation, we sample the intensities of the query image at pi and then assign it to the pixel coordinates in the reference view. After sampling, we have the frontalized view of the query image.


#### 3.1.1. Face Segmentation

Since the face of synthetic reference image appears in the center of image, after hard frontalization, all the synthetic frontal views of the query images also appear in the center of the image. This helps us to crop the face relevant regions of the frontal views to segment faces from the backgrounds. Following the idea in [10], we crop the 90 × 90 face relevant region of the frontalized image.


### 3.2. Support Vector-Guided Dictionary Learning

In Section2.2, we have introduced the general discriminative dictionary learning model. In this section, we will introduce how the constrains are imposed on the coefficients. Commonly, we can assess the discrimination through calculating the similarity of the coefficient vector pairs from the same class and the dissimilarity of the coefficient vector pairs from different classes. Therefore, we can indicate the discrimination capability by applying weight to the squared distance of the coefficient vector pairs vi, vj, and we have the following discrimination term L(V):(6)LV,wij=∑i,j∈Nvi−vj22wij.Suppose the weightswij can be parameterized as a function wijβ with respect to a variable β, and it satisfies the following three properties: awijβ=wijβ; bwijβ≥0 if vi and vj are from the same class si=sj, and wjiβ≤0 if vi and vj are from different classes si≠sj; c∑j=1nwijβ=0,∀i to balance the contributions of positive weights and negative weights, where si∈−1,1 represents the label of a two-class classification problem; all the coefficient vectors from the same class are labeled 1, and all the coefficient vectors from different classes are labeled -1.Denotewijβ=sisjβiβj; when ∑j=1nsiβj=0, the discrimination term can be rewritten as follows:(7)LV,wijβ=−2∑i,j∈NsisjβiβjviTvj=βTΩβ,where Ω is a negative semidefinite matrix; to obtain the extremum of β, we maximize equation (7):(8)β=argmaxββTΩβ+γβ,s.t.βi≥0,∀i,∑j=1nsjβj=0.By introducing a regularization termγβ=4∑i=1nβi, we have the parameterized formulation of DDL:(9)D,V=argminD,VT−DVF2+λ1vpp+λ2maxβ∑i,j∈Nvi−vj22wijβ+rβ,=argminD,VT−DVF2+λ1Vpp+λ2maxβ4∑i=1nβi−2∑i,j∈NsjsjβiβjviTvj,s.t.βi≥0,∀i,∑j=1nsiβj=0.From the settings ofwijβ, we can see that the problem on weight assignment is actually the Lagrange dual form of linear support vector machines (SVM) [19]. We impose an additional constraint βi≤1/2θ on β for all i to further reduce the effect of outliers, and θ is a constant parameter. By replacing the subproblem β with its primal SVM, we have the following support vector-guided dictionary learning (SVGDL) model:(10)D,V,u,b=argminD,V,u,bT−DVF2+λ1Vpp+2λ2LV,s,u,b,and LV,s,u,b is defined as follows:(11)LV,s,u,b=u22+θ∑i=1nℓvi,si,u,b,where u is the normal to the hyperplane of SVM, b is the corresponding bias, s=s1,s2,…,sn is the label vector, and ℓvi,si,u,b is the hinge loss function. The solution of u,b can be represented as the linear combination of a few coefficient vectors (support vectors), which means we only assign nonzero weights for pairwise support vectors.For multiclass classification, we simply adopt one-vs-all strategy by learningC hyperplanes U=u1,u2,…,uC and corresponding bias b=b1,b2,…,bC and formulate SVGDL as follows:(12)D,V,u,b=argminD,V,u,bT−DVF2+λ1V22+2λ2∑C=1CLV,s,u,b,where Sc=S1c,S2c,…,Snc if S1c=1, and otherwise S1c=−1.


#### 3.2.1. Optimization

The objective function 12 is not a convex optimization problem because there are four variablesD, V, u,b. Therefore, we divide three subprocedures: (a) computing u,b by fixing D and V; (b) updating V by fixing D and u,b; and (c) updating D by fixing V and u,b. The details are as follows:Step 1 (initializeD and V and u,b): initialize each class of the dictionary D as the training samples after feature extraction and WPCA transformation with unit ℓ2 norm. We use zero matrices and zeros vectors to initialize V and u,b.Step 2 (fixD and V and solve U,b): the minimization of u,b can be expressed as a multiclass linear SVM problem. Yang et al. [20] proposed a gradient-based method to learn uc and bc one by one.Step 3 (fixD and u,b and update V): we optimize the coefficient matrix V column by column, and we use the expression(13)vi=argminvit^i−Dvi22+λ1vi22+2λ2∑C=1Cℓvi,sic,uc,bc,to formulate the optimization of eachvi.Step 4 (fixV and u,b and update D): the optimization of dictionary D can be expressed as(14)D=argminDT−DVF2,s.t.dk2≤1,∀k∈1,2,…,K.Step 5: go back to Step 2 until the dictionary elements no longer change in two iterations or reach the maximum iteration number.


#### 3.2.2. Classification

After optimization, we have the learned dictionaryD and the SVM classifier u,b. For a test image, after feature extraction and Whitening PCA, we have its feature vector t∧. We use a matrix P^=DTD+λ1I−1DT to further project t:(15)t^¯=P^t^.Then, we use theC linear classifier uc,bc to predict the label of t∧∼:(16)label=argmaxc∈1,2,…,CucTv+bc.


## 3.1. Face Frontalization Method

Given a query image IQ and its landmarks, we can synthesis its frontal view. The following are the detailed steps for face frontalization. By specifying a projection matrix [18] CM=AMRMTM, where matrix AM denotes the intrinsic parameters and RMTM denotes the extrinsic parameters consisting of a rotation matrix RM and a translation vector TM, we can produce a synthetic view of the textured 3D face model. In this paper, we select an extrinsic matrix CM to produce the frontal view IR of the 3D face model, as shown in Figure 2. This frontal view IR will be used as our reference image as well as the reference coordinate system next.Figure 2
The synthetic reference face of the 3D face model.During the process of producing reference view, for each pixelp′ in the reference view, we store its corresponding 3D point coordinates P=X,Y,ZT located on the 3D face model. The correspondence between P and p′ can be obtained by(4)p′∼CMP.We denoteP=X,Y,ZT as the facial feature points of the query image IQ, and p′=xi′,yi′T are the same facial feature points of the reference view IR. From equation (4), we have the 3D coordinate point Pi=Xi,Yi,ZiT corresponding to the facial feature pi=x,iyiT of the query image. This provides the correspondences piT,PiT=xi,yi,Xi,Yi,Zi between the query image and the 3D face model; thus, we can estimate the projection matrix MQ=AQRQTQ, which is approximately the one used to capture the query image.With the projection matrix MQ, by using the expression,(5)p∼MQP.We have the correspondences between each pixel of the query image and each 3D coordinate of the 3D face model. Therefore, we have correspondences between each pixel of the query image and each pixel of the reference face. By using bilinear interpolation, we sample the intensities of the query image at pi and then assign it to the pixel coordinates in the reference view. After sampling, we have the frontalized view of the query image.


### 3.1.1. Face Segmentation

Since the face of synthetic reference image appears in the center of image, after hard frontalization, all the synthetic frontal views of the query images also appear in the center of the image. This helps us to crop the face relevant regions of the frontal views to segment faces from the backgrounds. Following the idea in [10], we crop the 90 × 90 face relevant region of the frontalized image.


## 3.1.1. Face Segmentation

Since the face of synthetic reference image appears in the center of image, after hard frontalization, all the synthetic frontal views of the query images also appear in the center of the image. This helps us to crop the face relevant regions of the frontal views to segment faces from the backgrounds. Following the idea in [10], we crop the 90 × 90 face relevant region of the frontalized image.


## 3.2. Support Vector-Guided Dictionary Learning

In Section2.2, we have introduced the general discriminative dictionary learning model. In this section, we will introduce how the constrains are imposed on the coefficients. Commonly, we can assess the discrimination through calculating the similarity of the coefficient vector pairs from the same class and the dissimilarity of the coefficient vector pairs from different classes. Therefore, we can indicate the discrimination capability by applying weight to the squared distance of the coefficient vector pairs vi, vj, and we have the following discrimination term L(V):(6)LV,wij=∑i,j∈Nvi−vj22wij.Suppose the weightswij can be parameterized as a function wijβ with respect to a variable β, and it satisfies the following three properties: awijβ=wijβ; bwijβ≥0 if vi and vj are from the same class si=sj, and wjiβ≤0 if vi and vj are from different classes si≠sj; c∑j=1nwijβ=0,∀i to balance the contributions of positive weights and negative weights, where si∈−1,1 represents the label of a two-class classification problem; all the coefficient vectors from the same class are labeled 1, and all the coefficient vectors from different classes are labeled -1.Denotewijβ=sisjβiβj; when ∑j=1nsiβj=0, the discrimination term can be rewritten as follows:(7)LV,wijβ=−2∑i,j∈NsisjβiβjviTvj=βTΩβ,where Ω is a negative semidefinite matrix; to obtain the extremum of β, we maximize equation (7):(8)β=argmaxββTΩβ+γβ,s.t.βi≥0,∀i,∑j=1nsjβj=0.By introducing a regularization termγβ=4∑i=1nβi, we have the parameterized formulation of DDL:(9)D,V=argminD,VT−DVF2+λ1vpp+λ2maxβ∑i,j∈Nvi−vj22wijβ+rβ,=argminD,VT−DVF2+λ1Vpp+λ2maxβ4∑i=1nβi−2∑i,j∈NsjsjβiβjviTvj,s.t.βi≥0,∀i,∑j=1nsiβj=0.From the settings ofwijβ, we can see that the problem on weight assignment is actually the Lagrange dual form of linear support vector machines (SVM) [19]. We impose an additional constraint βi≤1/2θ on β for all i to further reduce the effect of outliers, and θ is a constant parameter. By replacing the subproblem β with its primal SVM, we have the following support vector-guided dictionary learning (SVGDL) model:(10)D,V,u,b=argminD,V,u,bT−DVF2+λ1Vpp+2λ2LV,s,u,b,and LV,s,u,b is defined as follows:(11)LV,s,u,b=u22+θ∑i=1nℓvi,si,u,b,where u is the normal to the hyperplane of SVM, b is the corresponding bias, s=s1,s2,…,sn is the label vector, and ℓvi,si,u,b is the hinge loss function. The solution of u,b can be represented as the linear combination of a few coefficient vectors (support vectors), which means we only assign nonzero weights for pairwise support vectors.For multiclass classification, we simply adopt one-vs-all strategy by learningC hyperplanes U=u1,u2,…,uC and corresponding bias b=b1,b2,…,bC and formulate SVGDL as follows:(12)D,V,u,b=argminD,V,u,bT−DVF2+λ1V22+2λ2∑C=1CLV,s,u,b,where Sc=S1c,S2c,…,Snc if S1c=1, and otherwise S1c=−1.


### 3.2.1. Optimization

The objective function 12 is not a convex optimization problem because there are four variablesD, V, u,b. Therefore, we divide three subprocedures: (a) computing u,b by fixing D and V; (b) updating V by fixing D and u,b; and (c) updating D by fixing V and u,b. The details are as follows:Step 1 (initializeD and V and u,b): initialize each class of the dictionary D as the training samples after feature extraction and WPCA transformation with unit ℓ2 norm. We use zero matrices and zeros vectors to initialize V and u,b.Step 2 (fixD and V and solve U,b): the minimization of u,b can be expressed as a multiclass linear SVM problem. Yang et al. [20] proposed a gradient-based method to learn uc and bc one by one.Step 3 (fixD and u,b and update V): we optimize the coefficient matrix V column by column, and we use the expression(13)vi=argminvit^i−Dvi22+λ1vi22+2λ2∑C=1Cℓvi,sic,uc,bc,to formulate the optimization of eachvi.Step 4 (fixV and u,b and update D): the optimization of dictionary D can be expressed as(14)D=argminDT−DVF2,s.t.dk2≤1,∀k∈1,2,…,K.Step 5: go back to Step 2 until the dictionary elements no longer change in two iterations or reach the maximum iteration number.


### 3.2.2. Classification

After optimization, we have the learned dictionaryD and the SVM classifier u,b. For a test image, after feature extraction and Whitening PCA, we have its feature vector t∧. We use a matrix P^=DTD+λ1I−1DT to further project t:(15)t^¯=P^t^.Then, we use theC linear classifier uc,bc to predict the label of t∧∼:(16)label=argmaxc∈1,2,…,CucTv+bc.


## 3.2.1. Optimization

The objective function 12 is not a convex optimization problem because there are four variablesD, V, u,b. Therefore, we divide three subprocedures: (a) computing u,b by fixing D and V; (b) updating V by fixing D and u,b; and (c) updating D by fixing V and u,b. The details are as follows:Step 1 (initializeD and V and u,b): initialize each class of the dictionary D as the training samples after feature extraction and WPCA transformation with unit ℓ2 norm. We use zero matrices and zeros vectors to initialize V and u,b.Step 2 (fixD and V and solve U,b): the minimization of u,b can be expressed as a multiclass linear SVM problem. Yang et al. [20] proposed a gradient-based method to learn uc and bc one by one.Step 3 (fixD and u,b and update V): we optimize the coefficient matrix V column by column, and we use the expression(13)vi=argminvit^i−Dvi22+λ1vi22+2λ2∑C=1Cℓvi,sic,uc,bc,to formulate the optimization of eachvi.Step 4 (fixV and u,b and update D): the optimization of dictionary D can be expressed as(14)D=argminDT−DVF2,s.t.dk2≤1,∀k∈1,2,…,K.Step 5: go back to Step 2 until the dictionary elements no longer change in two iterations or reach the maximum iteration number.


## 3.2.2. Classification

After optimization, we have the learned dictionaryD and the SVM classifier u,b. For a test image, after feature extraction and Whitening PCA, we have its feature vector t∧. We use a matrix P^=DTD+λ1I−1DT to further project t:(15)t^¯=P^t^.Then, we use theC linear classifier uc,bc to predict the label of t∧∼:(16)label=argmaxc∈1,2,…,CucTv+bc.


## 4. Experiments

Labeled Faces in the Wild (LFW [21]) database is one of the main databases for face images. This database is constructed for studying face recognition problems under unconstrained environment. All the face images are collected from the Internet; therefore, this database shows dramatic variations in illuminations, expressions, skin shades, age groups, and backgrounds. There are more than 13,000 images in this database and nearly 5749 individuals are contained.In this database, the number of face images of each individual is different, for example, 4069 individuals have only one image, while other 1680 individuals have two or more images. Thus, this database is more suitable for face verification rather than face identification. To evaluate our proposed method on this database, we need to rearrange the original LFW database. To ensure sufficient training samples, we choose the 158 individuals that have at least 10 images for experiments. We randomly select 10 images from each individual and construct a new sub-database denote as LFW-subdatabase. The original size of the images in LFW database is 250 × 250. As mentioned in Section3.1, we cropped the 90 × 90 pixels face relevant region for all the images in LFW-subdatabase.


### 4.1. Facial Feature Detection and Frontalization

Before producing the frontal view of the query image, we need to detect the facial features of the query image and the synthetic reference image IR. In recent years, many effective facial feature detection methods were proposed, such as Active Appearance Model (AAM) [22, 23], ZhuRamanan Method [24], Supervised Descent Method (SDM) [25],and Dlib library [26]. In this paper, considering both speed and accuracy, we choose two state-of-the-art methods, SDM [25] and Dlib library [26], to detect the facial features of the images.


#### 4.1.1. Dlib Library

Figure3 shows the distribution of the 68 facial features that is detected by the Dlib library; the features are mainly located on the eyes, eyebrows, nose, mouth, and jawline.Figure 3
The distribution of the 68 feature points detected by dlib.
(a)(b)(c)It is common that an unconstrained image has more than one face in the image, as shown in Figure4. To synthesis the frontal view of the subject, we need to provide the right facial features of the target subject, which means we need a preprocessing step to select the target face correctly. Suppose there is an image IQ that appears more than one face, by using face detector, we can have the position of each face (usually draw a rectangle on the face, as shown in Figure 5, which means we can calculate the size of the rectangles). In general, the target face usually occupies a relatively larger space with respect to the other faces in the image; thus, the face with bigger size can be regarded as the target face.Figure 4
Unconstrained images in the LFW database that have more than one face.
(a)(b)(c)(d)(e)Figure 5
Face frames detected and marked by different face detectors: the above images are detected by dlib face detector and the following images are detected by Viola–Jones face detector.However, from Figure5, it can be seen that the region of the target face drawn by Dlib library is sometimes smaller than other faces in the image (via calculation), while Viola–Jones algorithm [27] performs much better (via calculation). Therefore, we use Viola–Jones algorithm to draw the regions of the faces and sort the face regions by size to find the target face (the largest region). Then, we use Dlib to detect the facial features of the target face and store the feature points of the target face for frontalization. After frontalization, we cropped the 90 × 90 pixels face relevant region; thus, we get a new database consisted of frontalized and cropped face images, and we denote it “LFW3D-dlib” database.


#### 4.1.2. Supervised Descent Method (SDM)

Unlike Dlib library, the SDM method detects only 49 facial feature points (without the jawline), as shown in Figure6.Figure 6
The distribution of the 49 feature points detected by SDM.
(a)(b)(c)For SDM face detection method, we also sort the size of the rectangles for all the faces in one image and regard the face with bigger size as the target face. Then, we save the feature points of the target face for frontalization. After frontalization, we cropped the 90 × 90 pixels face relevant region, and we get a new database consisted of frontalized and cropped face images, and we denote it “LFW3D-sdm” database.


#### 4.1.3. Failure of Feature Detection

As introduced above, we need to provide the facial feature of a given image to synthesize its frontal view, and we evaluate the frontalization method on the LFW-subdatabase to generate the frontal views of the images. We know in this database that there are 158 individuals, and each individual has 10 images. However, during facial feature detection, some facial features (only a few number of images) may fail to be detected because of pose variations or other reasons, and thus we cannot produce their frontal views. To ensure that every individual has the same number (10 images per person) of images for experiments, we use their original nonfrontalized images (90 × 90 pixels face relevant region) to fill the vacant positions.


#### 4.1.4. Improvement

Out-of-plane rotation of the head can cause some facial features to be less visible than others, particularly those on the sides of the nose and head. In [10], the authors suggest using conditional soft symmetry to deal with the visibility problem. The process of conditional soft symmetry is borrowing appearances (exclude the eyes) from corresponding symmetric sides of the face where facial features are poorly visible due to pose changes. In this paper, based on LFW database, the authors produced a new database which consisted of frontalized, symmetrized, and cropped face images. We denote this database as “LFW3D-hassner” database.Although conditional symmetry can deal with the pose-related visibility problems to some extend, it can also introduce face structure problems. For example, if one side of the face is occluded, symmetry can replicate the occlusion, leaving the final result unrecognizable. Therefore, in this paper, we slightly modified the original frontalization method. Firstly, we did not apply the conditional soft symmetry process on the initial frontalized faces. Secondly, we not only consider SDM method but also consider the state-of-the-art detector Dlib Library.


### 4.2. Notations

In this section, we introduce the scheme of our identification system in detail. We consider several data preprocessing techniques: method illustration, database illustration, feature extraction, feature dimensionality reduction, parameter settings, and our experiment platform.


#### 4.2.1. Method Illustration

We proposed a frontalization-based support vector-guided dictionary learning (FSVGDL) method in this paper. In Section4.1, we selected two facial feature detection methods: Dlib and SDM for frontalization. Therefore, combining these two detection methods with our FSVGDL method, we have two combined methods named FSVGDL-sdm and FSVGDL-dlib, respectively, for face identification.


#### 4.2.2. Database Illustration

As introduced in Section4.1, after the frontalization process, we can generate two new databases consisted of frontalized and cropped face images. Therefore, we totally have four versions of LFW database in our experiments denoted as LFW-sub, LFW3D-hassner [10], LFW3D-sdm, and LFW3D-dlib, respectively, as shown in Figure 7. The details of the four databases are as follows:(1)
LFW-subdatabase: this database contains 158 individuals and each individual has 10 images. All the images are segmented from the backgrounds, and the size of each image is 90 × 90.(2)
LFW3D-hassner database: this database is provided by Hassner et al. [10], and all the images are frontalized and cropped into 90 × 90 pixels. We rearranged this database to get the face identification database. We choose the 158 individuals that have at least 10 images, and randomly selected 10 images per individual for experiment.(3)
LFW3D-sdm database: this database is produced by the FSVGDL-sdm method. It includes 1580 images taken from 158 individuals (10 images for each individual). The size of each image is 90 × 90.(4)
LFW3D-dlib database: this database is produced by the FSVGDL-dlib method. It includes 1580 images taken from 158 individuals (10 images for each individual). The size of each image is 90 × 90.Figure 7
The differences of the same person on the four databases. (a) LFW-sub, (b) LFW3D-hassner, (c) LFW3D-sdm, and (d) LFW3D-dlib.
(a)(b)(c)(d)


#### 4.2.3. Feature Extraction

In this paper, we select two methods to extract face features: local Gabor feature [4] and hand-craft LBP [3] descriptor which are less sensitive to expression and illumination changes.


#### 4.2.4. Feature Dimensionality Reduction

Directly taking facial feature vectors for leaning may cause computational problem. Therefore, we use WPCA to reduce the dimensionality of the facial feature vectors before learning. The transformation matrix of whitening is computed only from the training samples.


#### 4.2.5. Parameter Settings

For both the FSVGDL-sdm and FSVGDL-dlib methods, we set the constant parameterθ = 5, and there are two parameters λ1, λ2 to be tuned in the experiments. The parameters are evaluated by 5-fold crossvalidation, and we fix λ1=2×10−3 and λ2=1×10−6 for both methods in all the experiments.


#### 4.2.6. Platform

All the experiments are performed on a computer with a 5-core CPU, 8 GB RAM, and 64 bit operating system.


### 4.3. Effectiveness of FSVGDL Using Gabor Wavelet

We experiment five state-of-the-art methods SRC [11], CRC [12], SVDL [28], FDDL [16], and SVGDL [17] and our FSVGDL method on the LFW-subdatabase to study their performances of face identification. In this experiment, we use Gabor wavelet to extract the features from the face images. The dimension of the extracted Gabor feature is 4840 (5 × 8 × 11 × 11, with 5 scales and 8 orientations and the down sampling rate is 8). For the 158 individuals, in order to compare the performance of the proposed method with other methods with varying number of training samples, we randomly select τ images from each individual to construct the training set, and the rest images are used to construct the testing set. To show how our algorithm would benefit from more training samples, we set the value of τ to 4, 5, 6, and 7, respectively. To explore the relationship between feature dimensionality and identification accuracy, we use WPCA to vary the feature dimension. We test the identification performance in the reduced feature dimension of (250; 300; 350; 400; 450) for τ = 4, (300; 350; 400; 450; 500), fr τ = 5, (200, 300, 400, 500, 600) for τ = 6, and (200, 300, 400, 500, 600) for τ = 7, respectively.We design two subexperiments. (1) In this experiment, we perform experiments on LFW-subdatabase. (2) In this experiment, to show the effectiveness of our improved face frontalization method, we perform experiments on the LFW3D-hassner database for all the competing algorithms, and our FSVGDL method is still evaluated on the LFW-subdatabase:(1)
In this experiment, we evaluate all the algorithms on the LFW-subdatabase. Table1 lists the best identification accuracy and the corresponding feature dimension for all the algorithms with τ training samples per individual using Gabor wavelet. Figure 8 shows the plot of identification accuracy versus the different feature dimensions for all the algorithms with τ training samples using Gabor wavelet.From Table1, we can see that, for all value of τ, our FSVGDL outperforms all other competing algorithms. When the number of training samples per person increases, all the algorithms perform better.Forτ = 4, the proposed FSVGDL-dlib achieves 83.88% and FSVGDL-sdm achieves 83.12% when the feature dimension is 350, while the maximum identification accuracy of all the competing algorithms is 50.54%.Forτ = 5, the proposed FSVGDL-dlib achieves 89.15% when the feature dimension is 500, and the proposed FSVGDL-sdm achieves 87.45% when the feature dimension is 400. It is worth noting that the identification accuracy increases more than 4% for the proposed methods. While other competing algorithms still does not exceed 60%.Forτ = 6, it can be seen that the proposed FSVGDL-dlib method yields 92.65% when the feature dimension is 400 and is 30% higher than other competing algorithms.Forτ = 7, the identification accuracy increases a lot compared to τ = 4. We can see that FSVGDL-dlib achieves 94.34% when the feature dimension is 600.(2)
In this experiment, all the competing algorithms are performed on the LFW3D-hassner database to show the effectiveness of our improved frontalization method. In other words, we combined the original frontalization process in paper [10] with the competing algorithms to study the performance. We denote the original competing algorithms SRC, CRC, SVDL, FDDL, and SVGDL as SRC-hassner, CRC-hassner, FDDL-hassner, and SVGDL-hassner, respectively. Table 2 lists the best identification accuracy and the corresponding feature dimension for all algorithms with τ training samples per individual using Gabor wavelet. Figure 9 shows the plot of identification accuracy versus the different feature dimensions for all the algorithms with τ training samples using Gabor wavelet, respectively.Figure 8
Identification accuracy versus dimensions of feature subspace of all algorithms with varyingτ number of training samples per individual using Gabor wavelet. (a) τ = 4, (b) τ = 5, (c) τ = 6, and (d) τ = 7.
(a)(b)(c)(d)Figure 9
Identification accuracy versus dimensions of feature subspace of all algorithms with varyingτ number of training samples per individual using Gabor wavelet. (a) τ = 4, (b) τ = 5, (c) τ = 6, and (d) τ = 7.
(a)(b)(c)(d)Table 1
Comparisons of best identification accuracy and the corresponding feature dimension between FSVGDL and other algorithms on LFW database with varyingτ number of training samples using Gabor wavelet. Bold are the best performers for each value of τ.
Methodsτ = 4τ = 5τ = 6τ = 7SRC46.51% (450)51.11% (500)56.21% (600)62.75% (500)CRC44.34% (400)47.84% (500)52.29% (500)58.17% (500)SVDL46.51% (450)50.98% (500)56.21% (600)63.62% (500)FDDL48.37% (400)52.94% (400)58.82% (400)66.01% (500)SVGDL50.54% (400)53.86% (500)59.80% (500)66.88% (500)FSVGDL-sdm83.12% (350)87.45% (400)90.85% (500)92.59% (500)FSVGDL-dlib83.88% (350)89.15% (500)92.65% (400)94.34% (600)Table 2
Comparisons of best identification accuracy and the corresponding feature dimension between FSVGDL and other algorithms with varyingτ number of training samples using Gabor wavelet. Bold are the best performers for each value of τ.
Methodsτ = 4τ = 5τ = 6τ = 7SRC-hassner76.69% (450)82.09% (500)86.60% (600)88.45% (600)CRC-hassner73.42% (350)79.48% (500)82.19% (400)86.27% (600)SVDL-hassner76.91% (450)81.83% (500)87.09% (600)88.24% (600)FDDL-hassner80.61% (350)84.58% (400)88.24% (400)90.63% (500)SVGDL-hassner81.05% (400)84.97% (400)89.38% (400)90.20% (300)FSVGDL-sdm83.12% (400)86.67% (400)89.54% (400)93.46% (500)FSVGDL-dlib85.29% (350)88.37% (400)90.85% (400)95.64% (500)Comparing the results in Tables1 and 2, we can see the accuracy of all the competing algorithms increase more than 20%. This shows the effectiveness of the original frontalization process. However, even though the performance of the competing algorithms improved a lot, our method still achieves the best accuracy no matter how many training samples are used, which shows the effectiveness of our improvement to the original face frontalization method. As shown in Figure 8, it can be seen that when the feature dimension changes, the accuracy of the algorithms also changes.Forτ = 4, the proposed FSVGDL-sdm achieves 83.12% when the feature dimension is 400, and FSVGDL-dlib achieves 85.29% when the feature dimension is 350. The maximum identification accuracy of all the competing algorithms is 81.05%, which is still 2.07% lower than FSVGDL-sdm and 4.24% lower than FSVGDL-dlib.Forτ = 5, we can see that FSVGDL-sdm yields 86.67% and FSVGDL-dlib yields 88.37% when the feature dimension is 400. The identification accuracy of proposed methods is still higher than the maximum accuracy of all the competing algorithms.Forτ = 6, the proposed FSVGDL-sdm achieves 89.54% and FSVGDL-dlib yields 90.85% when the feature dimension is 400, which is slightly higher than the identification accuracy of SVGDL-hassner algorithm.Forτ = 7, it can be seen that, compared with the results when τ = 4, the identification accuracy of the proposed methods increases nearly 10%. This phenomenon indicates that the increasing of training samples can lead to performance gain.


### 4.4. Effectiveness of FSVGDL Using LBP Descriptor

We experiment five state-of-the-art methods SRC [11], CRC [12], SVDL [28], FDDL [16], and SVGDL [17] and our FSVGDL method to study the performance of face identification on the LFW-subdatabases. In this experiment, we use Local Binary Pattern (LBP) to extract the features from the face images. The feature dimension of LBP is 8496 (12 × 12 × 59), and the face is divided into 8 × 8 nonoverlapping blocks. Similar to experiment that uses Gabor wavelet, for the 158 individuals, in order to compare the performance of the proposed method with other methods with varying number of training samples, we randomly select τ images from each individual to construct the training set, and the rest images are used to construct the testing set. Similarly, we set the value of τ to 4, 5, 6, and 7 to show how our algorithm would benefit from more training samples, and by using WPCA, the feature dimension is reduced to (250, 300, 350, 400, 450) for τ = 4, (300, 350, 400, 450, 500) for τ = 5, (200, 300, 400, 500, 600) for τ = 6, and (200, 300, 400, 500, 600) for τ = 7, respectively. We also design two subexperiments. (1) In this experiment, we perform experiments on the LFW-subdatabase for all the algorithms. (2) In this experiment, to show the effectiveness of our improved frontalization process, we perform experiments on the LFW3D-hassner database for all the competing algorithms.(1) In this experiment, all the algorithms are evaluated on the LFW-subdatabase. Table3 lists the best identification accuracy and the corresponding feature dimension for all the algorithms with τ training samples per individual using LBP descriptor. Figure 10 shows the plot of identification accuracy versus the different feature dimensions for all the algorithms with τ training samples per individual using LBP descriptor.Table 3
Comparisons of best identification accuracy and the corresponding feature dimension between FSVGDL and other algorithms on LFW database with varyingτ number of training samples using LBP descriptor. Bold are the best performers for each value of τ.
Methodsτ = 4τ = 5τ = 6τ = 7SRC44.44% (450)47.84% (600)52.61% (600)57.52% (600)CRC41.07% (500)46.41% (500)51.96% (600)55.56% (600)SVDL44.34% (450)47.97% (600)52.78% (600)58.17% (600)FDDL43.25% (400)47.97% (600)52.12% (600)58.17% (400)SVGDL43.57% (400)47.45% (500)52.94% (600)57.73% (500)FSVGDL-sdm79.96% (500)85.10% (500)87.09% (300)91.94% (500)FSVGDL-dlib82.35% (500)87.32% (500)90.20% (600)91.50% (600)Figure 10
Identification accuracy versus dimensions of feature subspace of all algorithms with varyingτ number of training samples per individual using LBP descriptor. (a) τ = 4, (b) τ = 5, (c) τ = 6, and (d) τ = 7.
(a)(b)(c)(d)From Table3, by comparing the performance of all the algorithms, we can see the effectiveness of our method. The accuracy varies when the value of τ changes, and the more training samples are used, the better accuracy these algorithms achieve. We can see from Figure 10 that when the feature dimension changes, the accuracy of the algorithms also changes. Comparing the results in Tables 1 and 3, it can be seen that Gabor wavelet is more suitable for feature extraction. All the algorithms achieve better accuracy when using Gabor wavelet.Forτ = 4, the proposed FSVGDL-dlib achieves 82.35% and FSVGDL-sdm achieves 79.96% when the feature dimension is 500, which is 30% higher than all the competing algorithms.Forτ = 5, the proposed FSVGDL-dlib achieves 87.32% and FSVGDL-sdm achieves 85.10% when the feature dimension is 500, while the maximum identification accuracy of all the competing algorithms does not exceed 50%.Forτ = 6, we can see that the proposed FSVGDL-dlib method yields 90.20% when the feature dimension is 600, and FSVGDL-sdm yields 87.09% when the feature dimension is 300, while the maximum identification accuracy of all the competing algorithms is 52.94% with 600 feature dimension.Forτ = 7, the identification accuracy increases a lot compared to τ = 4. It can be seen that FSVGDL-dlib achieves 91.50% when the feature dimension is 600, and FSVGDL-sdm achieves 91.94% when the feature dimension is 500. It is worth noting that the identification accuracy increases about 10% for the all the algorithms comparing with the results when τ = 4.(2) In this experiment, all the competing algorithms are performed on the LFW3D-hassner database. Similarly, we denote the original competing algorithms SRC, CRC, SVDL, FDDL, and SVGDL as SRC-hassner, CRC-hassner, FDDL-hassner, and SVGDL-hassner, respectively. Table4 lists the best identification accuracy and the corresponding feature dimension for all algorithms with τ training samples per individual using Gabor wavelet. Figure 11 shows the plot of accuracy versus the different feature dimensions for six methods with τ training samples using Gabor wavelet, respectively.Table 4
Comparisons of best identification accuracy and the corresponding feature dimension between FSVGDL and other algorithms with varyingτ number of training samples using LBP descriptor. Bold are the best performers for each value of τ.
Methodsτ = 4τ = 5τ = 6τ = 7SRC-hassner78.43% (350)81.96% (600)84.80% (600)88.45% (600)CRC-hassner75.93% (500)79.22% (400)84.48% (300)87.36% (500)SVDL-hassner78.65% (350)81.96% (600)84.97% (600)88.67% (600)FDDL-hassner78.98% (450)81.96% (500)85.95% (400)89.32% (300)SVGDL-hassner79.52% (300)82.22% (600)86.76% (300)90.85% (300)FSVGDL-sdm82.90% (350)85.10% (300)90.36% (500)91.50% (400)FSVGDL-dlib85.51% (300)87.84% (400)89.38% (600)93.03% (500)Figure 11
Identification accuracy versus dimensions of feature subspace of all algorithms with varyingτ number of training samples per individual using LBP descriptor. (a) τ = 4, (b) τ = 5, (c) τ = 6, and (d) τ = 7.
(a)(b)(c)(d)From Tables3 and 4, it can be seen that the accuracy of all the competing algorithms increase more than 20%, which indicates the effectiveness of frontalization. The performance of our method is better than all other algorithms, which indicates the effectiveness of our improvement on face frontalization. It can be seen from Figure 10 that when the feature dimension changes, the accuracy of the algorithms also changes. Comparing the results in Tables 1 and 3, we can also see that the accuracy of all algorithms is higher when using Gabor wavelet.Forτ = 4, it can be seen that the proposed FSVGDL-dlib yields 85.51%, which is 2.61% higher than FSVGDL-sdm, while FSVGDL-sdm is 3.38% higher than the maximum identification accuracy of all the competing algorithms (79.52%).Forτ = 5, compared with the results using 4 training samples per person, we can see that the best accuracy of all the algorithms increased more than 2%. FSVGDL-dlib achieves 87.84% when the feature dimension is 400.Forτ = 6, FSVGDL-sdm yields 90.36% when the feature dimension is 500 which is slightly higher than FSVGDL-dlib. Compared with the results when τ = 5, the performance of all the algorithms also improved more than 2%.Forτ = 7, the algorithms show a significant improvement compared with the results when τ = 6, the proposed FSVGDL-dlib yields 93.03% when the feature dimension is 500 and FSVGDL-sdm achieves 91.50% when the feature dimension is 400.We can see from the results in Tables1–4 that compared with the results using Gabor wavelet, the performance of all the competing algorithms have dropped when using LBP descriptor, which indicates the superiority of Gabor wavelet. We also notice that when using Gabor wavelet, the performance of FSVGDL-dlib is consistently better than FSVGDL-sdm, while in the experiments that using LBP descriptor, FSVGDL-sdm sometimes outperforms FSVGDL-dlib.


### 4.5. Decision Fusion

Decision fusion generally leads to performance gain [29, 30]. In this experiment, we not only perform fusion on FSVGDL-sdm and FSVGDL-dlib methods but also perform fusion on the Gabor and LBP descriptors. Therefore, we totally have four decisions for a test image. For each test image, we use voting (minority obeying majority) to make the final decision, which class does the test image belongs to.For all the competing algorithms, we do experiments on the LFW3D-sdm database and LFW3D-dlib database, respectively, wich means we apply our improved frontalization process on all the competing algorithms. With Gabor wavelet and LBP descriptor, each algorithm has four decisions: sdm-Gabor, sdm-LBP, dlib-Gabor, and dlib-LBP. We make decision fusion on these competing algorithms and our proposed FSVGDL withτ training samples per individual on different feature dimensions, respectively. Table 5 lists the best fusion accuracy and the corresponding feature dimension for all the algorithms with τ training samples per individual. Figure 12 shows the plot of fusion accuracy versus the different feature dimensions for all the algorithms with τ training samples per individual, respectively.Figure 12
Plot of fusion accuracy versus the different feature dimensions for all the algorithms withτ training samples per individual.
(a)(b)(c)(d)Even though the fusion method we applied (voting) is simple, the obtained results in Table5 show the effectiveness of decision fusion. When using 7 samples Table 5, per person for training and 3 samples for testing, our method FSVGDL achieves 97.17% after fusion, which is 1.75% higher than the maximum identification accuracy of all the competing algorithms.Table 5
The best fusion accuracy and the corresponding feature dimension for all the algorithms witht training samples per individual. Bold are the best performers for each value of τ.
Methodsτ = 4τ = 5τ = 6τ = 7SRC-fusion86.82% (500)90.46% (600)92.97% (600)95.21% (600)CRC-fusion82.57% (500)86.27% (500)89.22% (400)92.59% (500)SVDL-fusion86.82% (500)90.59% (500)92.97% (600)95.42% (600)FDDL-fusion88.13% (400)91.24% (500)93.79% (500)95.21% (500)FSVGDL-fusion88.24% (450)92.03% (300)94.77% (600)97.17% (600)


### 4.6. Further Analysis

In recent years, lots of deep learning-based approaches are proposed, and some of them (such as DeepID2+ [31], DeepID3 [32], and DSGNN [33]) have achieved a recognition rate of more than 99%. Despite this, deep learning-based approaches are usually data and computation hungry. Our proposed method is computationally efficient and does not need any specialized hardware. For example, it takes less than 1 second for one image to finish the whole frontalization process (including feature detection, frontalization, and segmentation). In addition to time advantage, our frontalization does not need any training compared to that of the DeepFace method in paper [34]. And in the experiment that uses Gabor wavelet (Section 4.3), when the feature dimension is 500, we use 7 image per person for training and 3 image per person for testing, and it takes only 30.6s to train the model and less than 0.0001s to classify one test image.


## 4.1. Facial Feature Detection and Frontalization

Before producing the frontal view of the query image, we need to detect the facial features of the query image and the synthetic reference image IR. In recent years, many effective facial feature detection methods were proposed, such as Active Appearance Model (AAM) [22, 23], ZhuRamanan Method [24], Supervised Descent Method (SDM) [25],and Dlib library [26]. In this paper, considering both speed and accuracy, we choose two state-of-the-art methods, SDM [25] and Dlib library [26], to detect the facial features of the images.


### 4.1.1. Dlib Library

Figure3 shows the distribution of the 68 facial features that is detected by the Dlib library; the features are mainly located on the eyes, eyebrows, nose, mouth, and jawline.Figure 3
The distribution of the 68 feature points detected by dlib.
(a)(b)(c)It is common that an unconstrained image has more than one face in the image, as shown in Figure4. To synthesis the frontal view of the subject, we need to provide the right facial features of the target subject, which means we need a preprocessing step to select the target face correctly. Suppose there is an image IQ that appears more than one face, by using face detector, we can have the position of each face (usually draw a rectangle on the face, as shown in Figure 5, which means we can calculate the size of the rectangles). In general, the target face usually occupies a relatively larger space with respect to the other faces in the image; thus, the face with bigger size can be regarded as the target face.Figure 4
Unconstrained images in the LFW database that have more than one face.
(a)(b)(c)(d)(e)Figure 5
Face frames detected and marked by different face detectors: the above images are detected by dlib face detector and the following images are detected by Viola–Jones face detector.However, from Figure5, it can be seen that the region of the target face drawn by Dlib library is sometimes smaller than other faces in the image (via calculation), while Viola–Jones algorithm [27] performs much better (via calculation). Therefore, we use Viola–Jones algorithm to draw the regions of the faces and sort the face regions by size to find the target face (the largest region). Then, we use Dlib to detect the facial features of the target face and store the feature points of the target face for frontalization. After frontalization, we cropped the 90 × 90 pixels face relevant region; thus, we get a new database consisted of frontalized and cropped face images, and we denote it “LFW3D-dlib” database.


### 4.1.2. Supervised Descent Method (SDM)

Unlike Dlib library, the SDM method detects only 49 facial feature points (without the jawline), as shown in Figure6.Figure 6
The distribution of the 49 feature points detected by SDM.
(a)(b)(c)For SDM face detection method, we also sort the size of the rectangles for all the faces in one image and regard the face with bigger size as the target face. Then, we save the feature points of the target face for frontalization. After frontalization, we cropped the 90 × 90 pixels face relevant region, and we get a new database consisted of frontalized and cropped face images, and we denote it “LFW3D-sdm” database.


### 4.1.3. Failure of Feature Detection

As introduced above, we need to provide the facial feature of a given image to synthesize its frontal view, and we evaluate the frontalization method on the LFW-subdatabase to generate the frontal views of the images. We know in this database that there are 158 individuals, and each individual has 10 images. However, during facial feature detection, some facial features (only a few number of images) may fail to be detected because of pose variations or other reasons, and thus we cannot produce their frontal views. To ensure that every individual has the same number (10 images per person) of images for experiments, we use their original nonfrontalized images (90 × 90 pixels face relevant region) to fill the vacant positions.


### 4.1.4. Improvement

Out-of-plane rotation of the head can cause some facial features to be less visible than others, particularly those on the sides of the nose and head. In [10], the authors suggest using conditional soft symmetry to deal with the visibility problem. The process of conditional soft symmetry is borrowing appearances (exclude the eyes) from corresponding symmetric sides of the face where facial features are poorly visible due to pose changes. In this paper, based on LFW database, the authors produced a new database which consisted of frontalized, symmetrized, and cropped face images. We denote this database as “LFW3D-hassner” database.Although conditional symmetry can deal with the pose-related visibility problems to some extend, it can also introduce face structure problems. For example, if one side of the face is occluded, symmetry can replicate the occlusion, leaving the final result unrecognizable. Therefore, in this paper, we slightly modified the original frontalization method. Firstly, we did not apply the conditional soft symmetry process on the initial frontalized faces. Secondly, we not only consider SDM method but also consider the state-of-the-art detector Dlib Library.


## 4.1.1. Dlib Library

Figure3 shows the distribution of the 68 facial features that is detected by the Dlib library; the features are mainly located on the eyes, eyebrows, nose, mouth, and jawline.Figure 3
The distribution of the 68 feature points detected by dlib.
(a)(b)(c)It is common that an unconstrained image has more than one face in the image, as shown in Figure4. To synthesis the frontal view of the subject, we need to provide the right facial features of the target subject, which means we need a preprocessing step to select the target face correctly. Suppose there is an image IQ that appears more than one face, by using face detector, we can have the position of each face (usually draw a rectangle on the face, as shown in Figure 5, which means we can calculate the size of the rectangles). In general, the target face usually occupies a relatively larger space with respect to the other faces in the image; thus, the face with bigger size can be regarded as the target face.Figure 4
Unconstrained images in the LFW database that have more than one face.
(a)(b)(c)(d)(e)Figure 5
Face frames detected and marked by different face detectors: the above images are detected by dlib face detector and the following images are detected by Viola–Jones face detector.However, from Figure5, it can be seen that the region of the target face drawn by Dlib library is sometimes smaller than other faces in the image (via calculation), while Viola–Jones algorithm [27] performs much better (via calculation). Therefore, we use Viola–Jones algorithm to draw the regions of the faces and sort the face regions by size to find the target face (the largest region). Then, we use Dlib to detect the facial features of the target face and store the feature points of the target face for frontalization. After frontalization, we cropped the 90 × 90 pixels face relevant region; thus, we get a new database consisted of frontalized and cropped face images, and we denote it “LFW3D-dlib” database.


## 4.1.2. Supervised Descent Method (SDM)

Unlike Dlib library, the SDM method detects only 49 facial feature points (without the jawline), as shown in Figure6.Figure 6
The distribution of the 49 feature points detected by SDM.
(a)(b)(c)For SDM face detection method, we also sort the size of the rectangles for all the faces in one image and regard the face with bigger size as the target face. Then, we save the feature points of the target face for frontalization. After frontalization, we cropped the 90 × 90 pixels face relevant region, and we get a new database consisted of frontalized and cropped face images, and we denote it “LFW3D-sdm” database.


## 4.1.3. Failure of Feature Detection

As introduced above, we need to provide the facial feature of a given image to synthesize its frontal view, and we evaluate the frontalization method on the LFW-subdatabase to generate the frontal views of the images. We know in this database that there are 158 individuals, and each individual has 10 images. However, during facial feature detection, some facial features (only a few number of images) may fail to be detected because of pose variations or other reasons, and thus we cannot produce their frontal views. To ensure that every individual has the same number (10 images per person) of images for experiments, we use their original nonfrontalized images (90 × 90 pixels face relevant region) to fill the vacant positions.


## 4.1.4. Improvement

Out-of-plane rotation of the head can cause some facial features to be less visible than others, particularly those on the sides of the nose and head. In [10], the authors suggest using conditional soft symmetry to deal with the visibility problem. The process of conditional soft symmetry is borrowing appearances (exclude the eyes) from corresponding symmetric sides of the face where facial features are poorly visible due to pose changes. In this paper, based on LFW database, the authors produced a new database which consisted of frontalized, symmetrized, and cropped face images. We denote this database as “LFW3D-hassner” database.Although conditional symmetry can deal with the pose-related visibility problems to some extend, it can also introduce face structure problems. For example, if one side of the face is occluded, symmetry can replicate the occlusion, leaving the final result unrecognizable. Therefore, in this paper, we slightly modified the original frontalization method. Firstly, we did not apply the conditional soft symmetry process on the initial frontalized faces. Secondly, we not only consider SDM method but also consider the state-of-the-art detector Dlib Library.


## 4.2. Notations

In this section, we introduce the scheme of our identification system in detail. We consider several data preprocessing techniques: method illustration, database illustration, feature extraction, feature dimensionality reduction, parameter settings, and our experiment platform.


### 4.2.1. Method Illustration

We proposed a frontalization-based support vector-guided dictionary learning (FSVGDL) method in this paper. In Section4.1, we selected two facial feature detection methods: Dlib and SDM for frontalization. Therefore, combining these two detection methods with our FSVGDL method, we have two combined methods named FSVGDL-sdm and FSVGDL-dlib, respectively, for face identification.


### 4.2.2. Database Illustration

As introduced in Section4.1, after the frontalization process, we can generate two new databases consisted of frontalized and cropped face images. Therefore, we totally have four versions of LFW database in our experiments denoted as LFW-sub, LFW3D-hassner [10], LFW3D-sdm, and LFW3D-dlib, respectively, as shown in Figure 7. The details of the four databases are as follows:(1)
LFW-subdatabase: this database contains 158 individuals and each individual has 10 images. All the images are segmented from the backgrounds, and the size of each image is 90 × 90.(2)
LFW3D-hassner database: this database is provided by Hassner et al. [10], and all the images are frontalized and cropped into 90 × 90 pixels. We rearranged this database to get the face identification database. We choose the 158 individuals that have at least 10 images, and randomly selected 10 images per individual for experiment.(3)
LFW3D-sdm database: this database is produced by the FSVGDL-sdm method. It includes 1580 images taken from 158 individuals (10 images for each individual). The size of each image is 90 × 90.(4)
LFW3D-dlib database: this database is produced by the FSVGDL-dlib method. It includes 1580 images taken from 158 individuals (10 images for each individual). The size of each image is 90 × 90.Figure 7
The differences of the same person on the four databases. (a) LFW-sub, (b) LFW3D-hassner, (c) LFW3D-sdm, and (d) LFW3D-dlib.
(a)(b)(c)(d)


### 4.2.3. Feature Extraction

In this paper, we select two methods to extract face features: local Gabor feature [4] and hand-craft LBP [3] descriptor which are less sensitive to expression and illumination changes.


### 4.2.4. Feature Dimensionality Reduction

Directly taking facial feature vectors for leaning may cause computational problem. Therefore, we use WPCA to reduce the dimensionality of the facial feature vectors before learning. The transformation matrix of whitening is computed only from the training samples.


### 4.2.5. Parameter Settings

For both the FSVGDL-sdm and FSVGDL-dlib methods, we set the constant parameterθ = 5, and there are two parameters λ1, λ2 to be tuned in the experiments. The parameters are evaluated by 5-fold crossvalidation, and we fix λ1=2×10−3 and λ2=1×10−6 for both methods in all the experiments.


### 4.2.6. Platform

All the experiments are performed on a computer with a 5-core CPU, 8 GB RAM, and 64 bit operating system.


## 4.2.1. Method Illustration

We proposed a frontalization-based support vector-guided dictionary learning (FSVGDL) method in this paper. In Section4.1, we selected two facial feature detection methods: Dlib and SDM for frontalization. Therefore, combining these two detection methods with our FSVGDL method, we have two combined methods named FSVGDL-sdm and FSVGDL-dlib, respectively, for face identification.


## 4.2.2. Database Illustration

As introduced in Section4.1, after the frontalization process, we can generate two new databases consisted of frontalized and cropped face images. Therefore, we totally have four versions of LFW database in our experiments denoted as LFW-sub, LFW3D-hassner [10], LFW3D-sdm, and LFW3D-dlib, respectively, as shown in Figure 7. The details of the four databases are as follows:(1)
LFW-subdatabase: this database contains 158 individuals and each individual has 10 images. All the images are segmented from the backgrounds, and the size of each image is 90 × 90.(2)
LFW3D-hassner database: this database is provided by Hassner et al. [10], and all the images are frontalized and cropped into 90 × 90 pixels. We rearranged this database to get the face identification database. We choose the 158 individuals that have at least 10 images, and randomly selected 10 images per individual for experiment.(3)
LFW3D-sdm database: this database is produced by the FSVGDL-sdm method. It includes 1580 images taken from 158 individuals (10 images for each individual). The size of each image is 90 × 90.(4)
LFW3D-dlib database: this database is produced by the FSVGDL-dlib method. It includes 1580 images taken from 158 individuals (10 images for each individual). The size of each image is 90 × 90.Figure 7
The differences of the same person on the four databases. (a) LFW-sub, (b) LFW3D-hassner, (c) LFW3D-sdm, and (d) LFW3D-dlib.
(a)(b)(c)(d)


## 4.2.3. Feature Extraction

In this paper, we select two methods to extract face features: local Gabor feature [4] and hand-craft LBP [3] descriptor which are less sensitive to expression and illumination changes.


## 4.2.4. Feature Dimensionality Reduction

Directly taking facial feature vectors for leaning may cause computational problem. Therefore, we use WPCA to reduce the dimensionality of the facial feature vectors before learning. The transformation matrix of whitening is computed only from the training samples.


## 4.2.5. Parameter Settings

For both the FSVGDL-sdm and FSVGDL-dlib methods, we set the constant parameterθ = 5, and there are two parameters λ1, λ2 to be tuned in the experiments. The parameters are evaluated by 5-fold crossvalidation, and we fix λ1=2×10−3 and λ2=1×10−6 for both methods in all the experiments.


## 4.2.6. Platform

All the experiments are performed on a computer with a 5-core CPU, 8 GB RAM, and 64 bit operating system.


## 4.3. Effectiveness of FSVGDL Using Gabor Wavelet

We experiment five state-of-the-art methods SRC [11], CRC [12], SVDL [28], FDDL [16], and SVGDL [17] and our FSVGDL method on the LFW-subdatabase to study their performances of face identification. In this experiment, we use Gabor wavelet to extract the features from the face images. The dimension of the extracted Gabor feature is 4840 (5 × 8 × 11 × 11, with 5 scales and 8 orientations and the down sampling rate is 8). For the 158 individuals, in order to compare the performance of the proposed method with other methods with varying number of training samples, we randomly select τ images from each individual to construct the training set, and the rest images are used to construct the testing set. To show how our algorithm would benefit from more training samples, we set the value of τ to 4, 5, 6, and 7, respectively. To explore the relationship between feature dimensionality and identification accuracy, we use WPCA to vary the feature dimension. We test the identification performance in the reduced feature dimension of (250; 300; 350; 400; 450) for τ = 4, (300; 350; 400; 450; 500), fr τ = 5, (200, 300, 400, 500, 600) for τ = 6, and (200, 300, 400, 500, 600) for τ = 7, respectively.We design two subexperiments. (1) In this experiment, we perform experiments on LFW-subdatabase. (2) In this experiment, to show the effectiveness of our improved face frontalization method, we perform experiments on the LFW3D-hassner database for all the competing algorithms, and our FSVGDL method is still evaluated on the LFW-subdatabase:(1)
In this experiment, we evaluate all the algorithms on the LFW-subdatabase. Table1 lists the best identification accuracy and the corresponding feature dimension for all the algorithms with τ training samples per individual using Gabor wavelet. Figure 8 shows the plot of identification accuracy versus the different feature dimensions for all the algorithms with τ training samples using Gabor wavelet.From Table1, we can see that, for all value of τ, our FSVGDL outperforms all other competing algorithms. When the number of training samples per person increases, all the algorithms perform better.Forτ = 4, the proposed FSVGDL-dlib achieves 83.88% and FSVGDL-sdm achieves 83.12% when the feature dimension is 350, while the maximum identification accuracy of all the competing algorithms is 50.54%.Forτ = 5, the proposed FSVGDL-dlib achieves 89.15% when the feature dimension is 500, and the proposed FSVGDL-sdm achieves 87.45% when the feature dimension is 400. It is worth noting that the identification accuracy increases more than 4% for the proposed methods. While other competing algorithms still does not exceed 60%.Forτ = 6, it can be seen that the proposed FSVGDL-dlib method yields 92.65% when the feature dimension is 400 and is 30% higher than other competing algorithms.Forτ = 7, the identification accuracy increases a lot compared to τ = 4. We can see that FSVGDL-dlib achieves 94.34% when the feature dimension is 600.(2)
In this experiment, all the competing algorithms are performed on the LFW3D-hassner database to show the effectiveness of our improved frontalization method. In other words, we combined the original frontalization process in paper [10] with the competing algorithms to study the performance. We denote the original competing algorithms SRC, CRC, SVDL, FDDL, and SVGDL as SRC-hassner, CRC-hassner, FDDL-hassner, and SVGDL-hassner, respectively. Table 2 lists the best identification accuracy and the corresponding feature dimension for all algorithms with τ training samples per individual using Gabor wavelet. Figure 9 shows the plot of identification accuracy versus the different feature dimensions for all the algorithms with τ training samples using Gabor wavelet, respectively.Figure 8
Identification accuracy versus dimensions of feature subspace of all algorithms with varyingτ number of training samples per individual using Gabor wavelet. (a) τ = 4, (b) τ = 5, (c) τ = 6, and (d) τ = 7.
(a)(b)(c)(d)Figure 9
Identification accuracy versus dimensions of feature subspace of all algorithms with varyingτ number of training samples per individual using Gabor wavelet. (a) τ = 4, (b) τ = 5, (c) τ = 6, and (d) τ = 7.
(a)(b)(c)(d)Table 1
Comparisons of best identification accuracy and the corresponding feature dimension between FSVGDL and other algorithms on LFW database with varyingτ number of training samples using Gabor wavelet. Bold are the best performers for each value of τ.
Methodsτ = 4τ = 5τ = 6τ = 7SRC46.51% (450)51.11% (500)56.21% (600)62.75% (500)CRC44.34% (400)47.84% (500)52.29% (500)58.17% (500)SVDL46.51% (450)50.98% (500)56.21% (600)63.62% (500)FDDL48.37% (400)52.94% (400)58.82% (400)66.01% (500)SVGDL50.54% (400)53.86% (500)59.80% (500)66.88% (500)FSVGDL-sdm83.12% (350)87.45% (400)90.85% (500)92.59% (500)FSVGDL-dlib83.88% (350)89.15% (500)92.65% (400)94.34% (600)Table 2
Comparisons of best identification accuracy and the corresponding feature dimension between FSVGDL and other algorithms with varyingτ number of training samples using Gabor wavelet. Bold are the best performers for each value of τ.
Methodsτ = 4τ = 5τ = 6τ = 7SRC-hassner76.69% (450)82.09% (500)86.60% (600)88.45% (600)CRC-hassner73.42% (350)79.48% (500)82.19% (400)86.27% (600)SVDL-hassner76.91% (450)81.83% (500)87.09% (600)88.24% (600)FDDL-hassner80.61% (350)84.58% (400)88.24% (400)90.63% (500)SVGDL-hassner81.05% (400)84.97% (400)89.38% (400)90.20% (300)FSVGDL-sdm83.12% (400)86.67% (400)89.54% (400)93.46% (500)FSVGDL-dlib85.29% (350)88.37% (400)90.85% (400)95.64% (500)Comparing the results in Tables1 and 2, we can see the accuracy of all the competing algorithms increase more than 20%. This shows the effectiveness of the original frontalization process. However, even though the performance of the competing algorithms improved a lot, our method still achieves the best accuracy no matter how many training samples are used, which shows the effectiveness of our improvement to the original face frontalization method. As shown in Figure 8, it can be seen that when the feature dimension changes, the accuracy of the algorithms also changes.Forτ = 4, the proposed FSVGDL-sdm achieves 83.12% when the feature dimension is 400, and FSVGDL-dlib achieves 85.29% when the feature dimension is 350. The maximum identification accuracy of all the competing algorithms is 81.05%, which is still 2.07% lower than FSVGDL-sdm and 4.24% lower than FSVGDL-dlib.Forτ = 5, we can see that FSVGDL-sdm yields 86.67% and FSVGDL-dlib yields 88.37% when the feature dimension is 400. The identification accuracy of proposed methods is still higher than the maximum accuracy of all the competing algorithms.Forτ = 6, the proposed FSVGDL-sdm achieves 89.54% and FSVGDL-dlib yields 90.85% when the feature dimension is 400, which is slightly higher than the identification accuracy of SVGDL-hassner algorithm.Forτ = 7, it can be seen that, compared with the results when τ = 4, the identification accuracy of the proposed methods increases nearly 10%. This phenomenon indicates that the increasing of training samples can lead to performance gain.


## 4.4. Effectiveness of FSVGDL Using LBP Descriptor

We experiment five state-of-the-art methods SRC [11], CRC [12], SVDL [28], FDDL [16], and SVGDL [17] and our FSVGDL method to study the performance of face identification on the LFW-subdatabases. In this experiment, we use Local Binary Pattern (LBP) to extract the features from the face images. The feature dimension of LBP is 8496 (12 × 12 × 59), and the face is divided into 8 × 8 nonoverlapping blocks. Similar to experiment that uses Gabor wavelet, for the 158 individuals, in order to compare the performance of the proposed method with other methods with varying number of training samples, we randomly select τ images from each individual to construct the training set, and the rest images are used to construct the testing set. Similarly, we set the value of τ to 4, 5, 6, and 7 to show how our algorithm would benefit from more training samples, and by using WPCA, the feature dimension is reduced to (250, 300, 350, 400, 450) for τ = 4, (300, 350, 400, 450, 500) for τ = 5, (200, 300, 400, 500, 600) for τ = 6, and (200, 300, 400, 500, 600) for τ = 7, respectively. We also design two subexperiments. (1) In this experiment, we perform experiments on the LFW-subdatabase for all the algorithms. (2) In this experiment, to show the effectiveness of our improved frontalization process, we perform experiments on the LFW3D-hassner database for all the competing algorithms.(1) In this experiment, all the algorithms are evaluated on the LFW-subdatabase. Table3 lists the best identification accuracy and the corresponding feature dimension for all the algorithms with τ training samples per individual using LBP descriptor. Figure 10 shows the plot of identification accuracy versus the different feature dimensions for all the algorithms with τ training samples per individual using LBP descriptor.Table 3
Comparisons of best identification accuracy and the corresponding feature dimension between FSVGDL and other algorithms on LFW database with varyingτ number of training samples using LBP descriptor. Bold are the best performers for each value of τ.
Methodsτ = 4τ = 5τ = 6τ = 7SRC44.44% (450)47.84% (600)52.61% (600)57.52% (600)CRC41.07% (500)46.41% (500)51.96% (600)55.56% (600)SVDL44.34% (450)47.97% (600)52.78% (600)58.17% (600)FDDL43.25% (400)47.97% (600)52.12% (600)58.17% (400)SVGDL43.57% (400)47.45% (500)52.94% (600)57.73% (500)FSVGDL-sdm79.96% (500)85.10% (500)87.09% (300)91.94% (500)FSVGDL-dlib82.35% (500)87.32% (500)90.20% (600)91.50% (600)Figure 10
Identification accuracy versus dimensions of feature subspace of all algorithms with varyingτ number of training samples per individual using LBP descriptor. (a) τ = 4, (b) τ = 5, (c) τ = 6, and (d) τ = 7.
(a)(b)(c)(d)From Table3, by comparing the performance of all the algorithms, we can see the effectiveness of our method. The accuracy varies when the value of τ changes, and the more training samples are used, the better accuracy these algorithms achieve. We can see from Figure 10 that when the feature dimension changes, the accuracy of the algorithms also changes. Comparing the results in Tables 1 and 3, it can be seen that Gabor wavelet is more suitable for feature extraction. All the algorithms achieve better accuracy when using Gabor wavelet.Forτ = 4, the proposed FSVGDL-dlib achieves 82.35% and FSVGDL-sdm achieves 79.96% when the feature dimension is 500, which is 30% higher than all the competing algorithms.Forτ = 5, the proposed FSVGDL-dlib achieves 87.32% and FSVGDL-sdm achieves 85.10% when the feature dimension is 500, while the maximum identification accuracy of all the competing algorithms does not exceed 50%.Forτ = 6, we can see that the proposed FSVGDL-dlib method yields 90.20% when the feature dimension is 600, and FSVGDL-sdm yields 87.09% when the feature dimension is 300, while the maximum identification accuracy of all the competing algorithms is 52.94% with 600 feature dimension.Forτ = 7, the identification accuracy increases a lot compared to τ = 4. It can be seen that FSVGDL-dlib achieves 91.50% when the feature dimension is 600, and FSVGDL-sdm achieves 91.94% when the feature dimension is 500. It is worth noting that the identification accuracy increases about 10% for the all the algorithms comparing with the results when τ = 4.(2) In this experiment, all the competing algorithms are performed on the LFW3D-hassner database. Similarly, we denote the original competing algorithms SRC, CRC, SVDL, FDDL, and SVGDL as SRC-hassner, CRC-hassner, FDDL-hassner, and SVGDL-hassner, respectively. Table4 lists the best identification accuracy and the corresponding feature dimension for all algorithms with τ training samples per individual using Gabor wavelet. Figure 11 shows the plot of accuracy versus the different feature dimensions for six methods with τ training samples using Gabor wavelet, respectively.Table 4
Comparisons of best identification accuracy and the corresponding feature dimension between FSVGDL and other algorithms with varyingτ number of training samples using LBP descriptor. Bold are the best performers for each value of τ.
Methodsτ = 4τ = 5τ = 6τ = 7SRC-hassner78.43% (350)81.96% (600)84.80% (600)88.45% (600)CRC-hassner75.93% (500)79.22% (400)84.48% (300)87.36% (500)SVDL-hassner78.65% (350)81.96% (600)84.97% (600)88.67% (600)FDDL-hassner78.98% (450)81.96% (500)85.95% (400)89.32% (300)SVGDL-hassner79.52% (300)82.22% (600)86.76% (300)90.85% (300)FSVGDL-sdm82.90% (350)85.10% (300)90.36% (500)91.50% (400)FSVGDL-dlib85.51% (300)87.84% (400)89.38% (600)93.03% (500)Figure 11
Identification accuracy versus dimensions of feature subspace of all algorithms with varyingτ number of training samples per individual using LBP descriptor. (a) τ = 4, (b) τ = 5, (c) τ = 6, and (d) τ = 7.
(a)(b)(c)(d)From Tables3 and 4, it can be seen that the accuracy of all the competing algorithms increase more than 20%, which indicates the effectiveness of frontalization. The performance of our method is better than all other algorithms, which indicates the effectiveness of our improvement on face frontalization. It can be seen from Figure 10 that when the feature dimension changes, the accuracy of the algorithms also changes. Comparing the results in Tables 1 and 3, we can also see that the accuracy of all algorithms is higher when using Gabor wavelet.Forτ = 4, it can be seen that the proposed FSVGDL-dlib yields 85.51%, which is 2.61% higher than FSVGDL-sdm, while FSVGDL-sdm is 3.38% higher than the maximum identification accuracy of all the competing algorithms (79.52%).Forτ = 5, compared with the results using 4 training samples per person, we can see that the best accuracy of all the algorithms increased more than 2%. FSVGDL-dlib achieves 87.84% when the feature dimension is 400.Forτ = 6, FSVGDL-sdm yields 90.36% when the feature dimension is 500 which is slightly higher than FSVGDL-dlib. Compared with the results when τ = 5, the performance of all the algorithms also improved more than 2%.Forτ = 7, the algorithms show a significant improvement compared with the results when τ = 6, the proposed FSVGDL-dlib yields 93.03% when the feature dimension is 500 and FSVGDL-sdm achieves 91.50% when the feature dimension is 400.We can see from the results in Tables1–4 that compared with the results using Gabor wavelet, the performance of all the competing algorithms have dropped when using LBP descriptor, which indicates the superiority of Gabor wavelet. We also notice that when using Gabor wavelet, the performance of FSVGDL-dlib is consistently better than FSVGDL-sdm, while in the experiments that using LBP descriptor, FSVGDL-sdm sometimes outperforms FSVGDL-dlib.


## 4.5. Decision Fusion

Decision fusion generally leads to performance gain [29, 30]. In this experiment, we not only perform fusion on FSVGDL-sdm and FSVGDL-dlib methods but also perform fusion on the Gabor and LBP descriptors. Therefore, we totally have four decisions for a test image. For each test image, we use voting (minority obeying majority) to make the final decision, which class does the test image belongs to.For all the competing algorithms, we do experiments on the LFW3D-sdm database and LFW3D-dlib database, respectively, wich means we apply our improved frontalization process on all the competing algorithms. With Gabor wavelet and LBP descriptor, each algorithm has four decisions: sdm-Gabor, sdm-LBP, dlib-Gabor, and dlib-LBP. We make decision fusion on these competing algorithms and our proposed FSVGDL withτ training samples per individual on different feature dimensions, respectively. Table 5 lists the best fusion accuracy and the corresponding feature dimension for all the algorithms with τ training samples per individual. Figure 12 shows the plot of fusion accuracy versus the different feature dimensions for all the algorithms with τ training samples per individual, respectively.Figure 12
Plot of fusion accuracy versus the different feature dimensions for all the algorithms withτ training samples per individual.
(a)(b)(c)(d)Even though the fusion method we applied (voting) is simple, the obtained results in Table5 show the effectiveness of decision fusion. When using 7 samples Table 5, per person for training and 3 samples for testing, our method FSVGDL achieves 97.17% after fusion, which is 1.75% higher than the maximum identification accuracy of all the competing algorithms.Table 5
The best fusion accuracy and the corresponding feature dimension for all the algorithms witht training samples per individual. Bold are the best performers for each value of τ.
Methodsτ = 4τ = 5τ = 6τ = 7SRC-fusion86.82% (500)90.46% (600)92.97% (600)95.21% (600)CRC-fusion82.57% (500)86.27% (500)89.22% (400)92.59% (500)SVDL-fusion86.82% (500)90.59% (500)92.97% (600)95.42% (600)FDDL-fusion88.13% (400)91.24% (500)93.79% (500)95.21% (500)FSVGDL-fusion88.24% (450)92.03% (300)94.77% (600)97.17% (600)


## 4.6. Further Analysis

In recent years, lots of deep learning-based approaches are proposed, and some of them (such as DeepID2+ [31], DeepID3 [32], and DSGNN [33]) have achieved a recognition rate of more than 99%. Despite this, deep learning-based approaches are usually data and computation hungry. Our proposed method is computationally efficient and does not need any specialized hardware. For example, it takes less than 1 second for one image to finish the whole frontalization process (including feature detection, frontalization, and segmentation). In addition to time advantage, our frontalization does not need any training compared to that of the DeepFace method in paper [34]. And in the experiment that uses Gabor wavelet (Section 4.3), when the feature dimension is 500, we use 7 image per person for training and 3 image per person for testing, and it takes only 30.6s to train the model and less than 0.0001s to classify one test image.


## 5. Conclusions

In this paper, we proposed a frontalization-based support vector-guided dictionary learning (FSVGDL) method for unconstrained face identification. The key of FSVGDL is to produce the frontal view of the face image and assign different weights and different pairs of coding vectors adaptively to enhance the discriminative capability.Extensive experiments have been carried out on the LFW database. Five state-of-the-art algorithms SRC, CRC, SVDL, FDDL, and SVGDL are implemented to compare with the proposed FSVGDL method. The proposed method FSVGDL demonstrates its consistent superiority in its performance with respect to the state-of-the-art methods when considering different feature extraction methods. We also notice that the identification accuracy is better when using Gabor wavelet to extract features. The increasing number of training samples generally leads to the improvement of algorithm performance. The experiments of decision fusion show the effectiveness of fusion, and all the algorithms perform better after fusion. The proposed method yields 97.17% with 7 training samples per person with decision fusion. In the future, we will focus on finding an appropriate dimension of feature subspace to balance the computation cost and recognition accuracy.



---

*Source: 8817571-2020-11-03.xml*