# Fusing Two Kinds of Virtual Samples for Small Sample Face Recognition

Authors: Minna Qiu; Jian Zhang; Jiayan Yang; Liying Ye
Journal: Mathematical Problems in Engineering
 (2015)

---

## Abstract

Face recognition has become a very active field of biometrics. Different pictures of the same face might include various changes of expressions, poses, and illumination. However, a face recognition system usually suffers from the problem that nonsufficient training samples cannot convey these possible changes effectively. The main reason is that a system has only limited storage space and limited time to capture training samples. Many previous literatures ignored the problem of nonsufficient training samples. In this paper, we overcome the insufficiency of training sample size problem by fusing two kinds of virtual samples and the original samples to perform small sample face recognition. The two used kinds of virtual samples are mirror faces and symmetrical faces. Firstly, we transform the original face image to obtain mirror faces and symmetrical faces. Secondly, we fuse these two kinds of virtual samples to achieve the matching scores between the test sample and each class. Finally, we integrate the matching scores to get the final classification results. We compare the proposed method with the single virtual sample augment methods and the original representation-based classification. The experiments on various face databases show that the proposed scheme achieves the best accuracy among the representation-based classification methods.


---

## Body

## 1. Introduction

Face recognition [1–5] is paid more and more attention as a branch of biometrics [6–9]. Face images are usually unstable and a plenty of factors such as changes in facial expression, poses, light conditions (day/night, indoor/outdoor), coverings (mask, sunglasses, hair, beard, etc.), and the age adversely affect the performance of face recognition. In recent years, many researchers have made their efforts to address these challenges. For example, Beymer [10] proposed a view-based approach for recognizing faces under varying pose. The recognizer consists of two main stages, a geometrical alignment stage where the input is registered with the model views and a correlation stage for matching. Georghiades et al. [11] presented a generative appearance-based method for recognizing human faces under variation in lighting and viewpoint. Yang et al. [12] presented a novel robust sparse coding (RSC) model and an effective iteratively reweighted sparse coding (IRSC) algorithm for RSC to decrease the influence of various types of outliers. Wagner et al. [13] proposed a system for recognizing human faces from images taken under practical conditions that is conceptually simple, well motivated, and competitive with state-of-the-art recognition systems for access control scenarios. If we have sufficient and available training samples, which contain all the possible variations of the illumination, facial expression, and pose, we will obtain a high accuracy. However, in a practical application, the training samples always cannot convey sufficient variations of the illumination, facial expression, and pose [14–16]. Therefore, limited number of available training samples becomes a severe problem.For face recognition, fewer samples per person mean less laborious effort for collecting them and lower cost for storing and processing them. Unfortunately, the small size of training samples will bring about some challenges such as lower robustness. For nonsufficient training samples situation, previous literatures have proposed some approaches to generate new face images so that the size of training samples is enlarged [17–19]. These generated face images are called virtual samples. The virtual samples will contain the same pixels as the original samples. Suppose that the original image is X
∈
R
m
×
n; then the virtual samples can be represented as X
′
∈
R
m
×
n. For example, Tan et al. [20] attempted to provide a comprehensive survey of current researches on one sample per person face recognition problem. They evaluated some advantages and disadvantages of previous relevant methods. Xu et al. [21] assumed that the facial structure is symmetrical and proposed an approach to generate “symmetrical” face images and exploited both the original and “symmetrical” face images to recognize the subject. In [22] and [23], Xu et al. aimed at solving nonsymmetrical samples and misalignment problem. They proposed a method which exploits the mirror image of the face image to simulate possible variation. Su et al. [24] proposed to adapt the within-class and between-class scatter matrices computed from a generic training set to the persons to be identified by coupled linear representation method.If the training sample does not have a considerable size, the face recognition method using only the original face images will hardly obtain satisfactory accuracy. In this paper, the proposed method integrates the original training sample and its virtual samples to perform face recognition. The virtual samples include two parts: mirror face images and symmetrical face images. Each virtual face image reflects some possible change in pose and illumination of the original face image. The main advantages of mirror virtual training samples are that they can effectively overcome the problem of nonsymmetrical samples and misalignment problem [23]. The advantages of symmetrical faces are that they are different from the original face image but can really reflect some possible appearance of the face. Figure 1 shows the original samples, the corresponding virtual samples, and the test samples. The proposed scheme is not a simple combination of all the virtual samples and training samples, but a fusion of weights among the original training samples, the mirror virtual faces, and the symmetrical virtual faces. It is very important to select the superior fusion of weights; however, the details of selection process will be presented in Section 5.(a) The original training sample. (b) The corresponding mirror virtual sample. (c) The corresponding left symmetrical virtual samples. (d) The corresponding right symmetrical virtual samples. (e) The test sample from the corresponding class.
(a)
(b)
(c)
(d)
(e)To verify the effectiveness of our scheme, we apply it to several representation based classification (RBC) methods [25–28]. Some face recognition methods have been shown to be extremely brittle when they are faced with challenges such as alignment variation or minor occlusions. Many researchers have made their efforts to address the brittleness. The recently proposed sparse representation classification (SRC) method [29–31] can obtain a high accuracy in face recognition. In SRC, the testing image is represented as a sparse linear combination of the training samples and then the deviation between the test sample and the expression result of every class is used to perform classification. SRC has demonstrated striking recognition performance when facing noise such as occlusion or corruption. Relevant literatures [10] reveal that SRC performs well even if face samples were disturbed by 80 percent of random noises. In this paper, we conduct contrast experiments on some representative SRC methods, which include the collaborative representation classification (CRC) method [26], an improvement to the nearest neighbor classifier (INNC) method [27], a simple and fast representation based face recognition (SFRFR) method [28], and the two-phase test sample sparse representation (TPTSR) method [9].The contributions of the paper are as follows.(1)
The used virtual samples are composed of mirror virtual faces and symmetrical virtual faces. The simultaneous use of these two kinds of virtual samples gives good solution to the problem of nonsufficient sample problem.
(2)
The proposed method is not based on a single combination of all the virtual samples and training samples but is based on a weighted fusion of the original training samples, the mirror virtual faces, and the symmetrical virtual faces.The remainder of the paper is organized as follows. Section2 presents related works. Section 3 describes our proposed scheme. Section 4 shows the experimental results. Section 5 offers the conclusion.


## 2. Related Work

To verify the validity of our scheme, we will apply it to several representation based classification (RBC) methods [32, 33]. In this section, we present a brief introduction to the RBC method.Among RBC methods, the sparse representation classification (SRC) was almost the earliest proposed one. Later sparse representation methods based on thel
1-norm regularization [34, 35], l
2-norm regularization [36, 37], and l
2,1-norm regularization [38, 39] have also been proposed for face recognition. Among them, the l
2-norm minimization methods are widely used because of their lower computational complexity and good classification results. For example, Xu et al. [25] proposed a simple yet efficient face classification scheme, namely, collaborative representation based classification (CRC). The l
2-regularized CRC method exploits sparse linear equations to represent samples and divides the class specific representation residual by the l
2-norm “sparsity” to get the final classification result. In this paper, we conduct contrast experiments on CRC, INNC, SFRFR, and TPTSR which are all l
2-regularized RBC methods. We take CRC and TPTSR as an example to introduce the basic algorithm of RBC. The INNC method has the same equation and solution scheme as CRC but uses a simpler classifier. The SFRFR method has the same equation and solution scheme as TPTSR but only selects L nearest neighbors, each being from one class, for the test sample. We assume that there are c classes, and each class contains n training samples. Let x
1
,
…
,
x
N denote all training samples X, where N
=
c
*
n, and x
(
i
-
1
)
n
+
k stands for the kth training sample of ith subject. Let y denote the test sample. The CRC and TPTSR are summarized as Algorithms 1 and 2, respectively.Algorithm 1:The CRC algorithm.
(1) Normalize the columns ofX and y to have unit l
2-norm
(2) Solvey
=
X
A
IfX
T
X is singular, A
^
=
(
X
T
X
+
μ
I
)
-
1
X
T
y, where μ is a small
positive constant andI is the identity matrix.
IfX
T
X is not singular, A
^
=
(
X
T
X
)
-
1
X
T
y.
(3) Compute the regularized residuals
r
i
=
‖
y
-
X
i
⋅
A
^
i
‖
2
‖
A
^
i
‖
2
(4) Output the identity ofy as
Identity(
y
) = argmin
i
{
r
i
}Algorithm 2:The TPTSR algorithm.
(1) Normalize the columns ofX and y to have unit l
2-norm
(2) Solvey
=
X
A
IfX
T
X is singular, A
^
=
(
X
T
X
+
μ
I
)
-
1
X
T
y, where μ is a small
positive constant andI is the identity matrix.
IfX
T
X is not singular, A
^
=
(
X
T
X
)
-
1
X
T
y.
(3) Compute the deviation
e
i
=
‖
y
-
X
i
⋅
A
^
i
‖
2
(4) Output theM training samples that have the M greatest contributions
X
~
=
[
x
1
,
…
,
x
M
]
(5) Solvey
=
X
~
B
IfX
~
T
X
~ is singular, B
^
=
(
X
~
T
X
~
+
γ
I
)
-
1
X
~
T
y, where r is a small
positive constant andI is the identity matrix;
IfX
~
T
X
~ is not singular, B
^
=
(
X
~
T
X
~
)
-
1
X
~
T
y.
(6) Compute the deviation
g
r
=
‖
y
-
X
r
⋅
B
^
r
‖
2
(7) Output the identity ofy as
Identity(
y
)
=
argmin
r
{
g
r
}


## 3. The Proposed Scheme

In this section, we will introduce the proposed fused virtual samples augment method in detail.The proposed method uses a simple way to obtain more training samples and to improve the face recognition accuracy. The main steps are showed as follows. The first and second steps generate mirror training sample and “symmetrical face” training samples. These samples reflect possible variation of the face. The third step exploits original training samples for classification. The fourth, fifth, and sixth steps, respectively, exploit the mirror virtual training samples, the left symmetrical virtual training samples, and the right symmetrical virtual training samples for classification. The seventh step uses the score fusion for ultimate face recognition. Suppose that there arec classes and each class has n training samples; we present these steps as follows.Step 1.
Use original training sample to generate mirror training sample. Letx
i
∈
R
row
×
col be the ith training sample in the form of image matrix. Let x
i
mirror stand for the mirror training sample. The mirror virtual sample S of an arbitrary original training sample R is defined as S
(
p
,
q
)
=
R
(
p
,
col
-
q
+
1
), p
=
1
,
…
,
row, q
=
1
,
…
,
col. Row and col stand for the numbers of the rows and columns of R, respectively. S
(
p
,
q
) denotes the pixel located in the pth row and qth column of S. Figure 2 presents several original training samples in the ORL database and their corresponding mirror virtual training samples.The first row shows the original training samples in the ORL database. The second row shows the corresponding mirror virtual training samples.
(a)
(b)Step 2.
Use original training sample to generate the left and right symmetrical virtual training samples. Letx
i
left and x
i
right stand for the left and right symmetrical training samples generated from the ith training sample, respectively. The left symmetrical virtual image K
left of an arbitrary original training sample R is defined as K
left
p
,
m
=
R
p
,
m and K
left
p
,
q
=
R
p
,
col
/
2
-
(
q
-
col
/
2
-
1
), p
=
1
,
…
,
row, m
=
1
,
…
,
col
/
2, q
=
col
/
2
+
1
,
…
,
col. The right symmetrical virtual image K
right of an arbitrary original training sample R is defined as K
right
p
,
q
=
R
p
,
col
-
(
q
-
1
) and K
right
p
,
m
=
R
p
,
m, p
=
1
,
…
,
row, q
=
1
,
…
,
col
/
2, m
=
col
/
2
+
1
,
…
,
col. Figure 3 presents several original training samples in the ORL database and their corresponding symmetrical virtual training samples.The first row shows several original training samples in the ORL database. The second row shows the corresponding left symmetrical virtual training samples. The third row shows the corresponding right symmetrical virtual training samples.
(a)
(b)
(c)Step 3.
Use the original training samples to perform representation based method. Lets
j
1 denote the score of test sample z with respect to the jth class.Step 4.
Use the corresponding mirror virtual training samples to perform representation based method. Lets
j
2 denote the score of test sample z with respect to the jth class.Step 5.
Use the corresponding left symmetry virtual training samples to perform representation based method. Lets
j
3 denote the score of test sample z with respect to the jth class.Step 6.
Use the corresponding right symmetry virtual training samples to perform representation based method. Lets
j
4 denote the score of test sample z with respect to the jth class.Step 7.
Combine all scores obtained using Steps3, 4, 5, and 6 to conduct weighted score level fusion. For test samples, we use s
j
=
ω
1
s
j
1
+
ω
2
s
j
2
+
ω
3
s
j
3
+
ω
4
s
j
4 to calculate the ultimate score with respect to the test sample to z in the jth class. ω
1, ω
2, ω
3, and ω
4 represent the weights. Let ω
1
+
ω
2
+
ω
3
+
ω
4
=
1 and ω
1
≥
ω
2
+
ω
3
+
ω
4. Figure 4 shows the main steps of fusing virtual samples technique to do face recognition.Figure 4
Main steps of fusing virtual samples technique for face recognition.


## 4. Experimental Results

In this section, we evaluate the performance of our proposed scheme for face recognition. Then we make comprehensive analysis on the experimental results. In addition, we explain the selection of weights in detail and show the experimental comparison of different fusions of weights.


### 4.1. Data Sets

We conduct a number of experiments on the ORL [40], Yale [41], FERET [42], and FLW [43] databases. Figure 5 shows some original training samples from the ORL face database. Figure 6 shows some original training samples from the Yale face database. Figure 7 shows some original samples from the FERET face database. Figure 8 shows some original samples from the FLW face database. The database is introduced as follows.Figure 5
Several original training samples from the ORL face database.Figure 6
Several original training samples from the Yale face database.Figure 7
Several original training samples from the FERET face database.Figure 8
Several original training samples from the LFW face database.ORL. This data set contains 400 face images from 40 subjects, each providing ten images. The ORL database includes variations in facial expression (smiling/not smiling, open/closed eyes) and facial details. Each of the face images contains 32 × 32 pixels. The first 1, 2, and 3 images per person were selected for training and the remaining for testing.Yale. This data set contains 14 subjects, and each subject has 11 images. The Yale database includes variations in facial expression and facial details. Each of the face images contains 100 × 100 pixels. The first 1, 2, and 3 images per person were selected for training and the remaining for testing.FERET. This data set contains 200 subjects, and each subject has 7 images. Each face sample contains 40 × 40 pixels. FERET mainly includes change of illumination and change of expression. The first 1, 2, and 3 images per person were selected for training and the remaining for testing.LFW. This data set contains more than 13,000 images of faces collected from the web. Each face has been labeled with the name of the people pictured. 1680 of the people pictured have two or more distinct photos in the database. In the experiments, 860 images from 86 people were chosen. Each person has 10 images. The first 1, 2, and 3 images per person were selected for training and the remaining for testing.


### 4.2. Experimental Results and Analysis

Actually, in the beginning of studying this topic, we tried to use both the original training samples and their virtual training samples to do recognition, but the results are not satisfying. The basic steps of the method which combine the original training samples and the generated virtual images into a new training sample set for face recognition (COVNFR) are showed in Figure9.Figure 9
Basic steps of the COVNFR method for face recognition.We use the collaborative representation based classification (CRC) method to do the experiment in ORL, Yale, FERET, and FLW databases. The experimental results are showed in Table1.Table 1
Classification error rates of the COVNFR method.
Database
The number of training samples
1
2
3
CRC-original
0.3194
0.1656
0.1393
ORL
CRC-virtual training samples only
0.9306
0.9313
0.9643
CRC-COVNFR
0.3778
0.2719
0.2750
CRC-original
0.3733
0.1556
0.1083
Yale
CRC-virtual training samples only
0.9000
0.9333
0.9667
CRC-COVNFR
0.3533
0.1259
0.1000
CRC-original
0.5567
0.4160
0.5563
FERET
CRC-virtual training samples only
0.9592
0.8810
0.7937
CRC-COVNFR
0.6200
0.5290
0.6525
CRC-original
0.8437
0.7660
0.7641
FLW
CRC-virtual training samples only
0.9470
0.9608
0.9269
CRC-COVNFR
0.9031
0.8314
0.8306From Table1, we find that the COVNFR method has higher classification error rates than the original face recognition methods. The reasons why the COVNFR method cannot achieve a satisfying result are the following: if we use the virtual training samples only to do face recognition, all of the classification rates in Algorithm 1 are higher than 90% which means the virtual training sample cannot inherit all of the features. However, in the condition of limited training sample, only original training samples cannot conclude comprehensive features, while the virtual training samples inherit some features from the original training samples. Therefore, the use of weight combination to combine the original training samples and their virtual training samples not only adds feature information, but also regulates the effect of each part of training samples.Our scheme is based on a weighted fusion of the original training samples, the mirror virtual faces, and the symmetrical virtual faces which can be applied to lots of face recognition methods. In this paper, we conduct contrast experiments on some representative RBC methods, which include the collaborative representation classification (CRC) method [24], an improvement to the nearest neighbor classifier (INNC) method [25], a simple and fast representation based face recognition (SFRFR) method [26], and a two-phase test sample sparse representation (TPTSR) method [27]. From Tables 2, 3, 4, and 5, we compare some RBCs with different ways to process training samples. In Algorithm 1, “CRC-original” represents the original CRC method; “CRC-mirror face” represents integrating the original face image and its mirror sample to perform CRC method; “CRC-symmetrical face” represents integrating the original face image and its symmetrical sample to perform CRC method; “CRC-the proposed scheme” represents integrating the original face image and its fused virtual sample to perform CRC method. As can be seen from these four tables, the proposed scheme performs well when compared to all the other methods. For the LFW database, there are more complex variations in the face sample than other databases. However, the proposed method still achieves a satisfactory performance.Table 2
Classification error rates of different algorithms on the ORL database.
The number of training samples
1
2
3
CRC-original
0.3194
0.1656
0.1393
CRC-mirror face
0.2944
0.1469
0.1321
CRC-symmetrical face
0.3056
0.1500
0.1286
CRC-the proposed scheme
0.2944
0.1437
0.1179
INNC
0.3194
0.2125
0.2179
INNC-mirror face
0.2944
0.2125
0.2000
INNC-symmetrical face
0.3000
0.2094
0.2000
INNC-the proposed scheme
0.2944
0.1938
0.1750
SFRFR
0.3194
0.1969
0.1750
SFRFR-mirror face only
0.2944
0.1750
0.1250
SFRFR-symmetrical face only
0.3000
0.1656
0.1179
SFRFR-the proposed scheme
0.2944
0.1656
0.0929
TPTSR
0.2861
0.1250
0.1000
TPTSR-mirror face only
0.2722
0.1219
0.0893
TPTSR-symmetrical face only
0.2806
0.1094
0.0821
TPTSR-the proposed scheme
0.2667
0.1094
0.0750Table 3
Classification error rates of different algorithms on the Yale database.
The number of training samples
1
2
3
CRC-original
0.3733
0.1556
0.1083
CRC-mirror face
0.2800
0.1259
0.0333
CRC-symmetrical face
0.3067
0.1037
0.0500
CRC-the proposed scheme
0.2200
0.0889
0.0250
INNC
0.3733
0.2296
0.1833
INNC-mirror face
0.2800
0.1852
0.1250
INNC-symmetrical face
0.3067
0.2000
0.1583
INNC-the proposed scheme
0.2200
0.1630
0.1000
SFRFR
0.3733
0.2148
0.1417
SFRFR-mirror face only
0.2800
0.2000
0.0833
SFRFR-symmetrical face only
0.3067
0.1778
0.1167
SFRFR-the proposed scheme
0.2200
0.1333
0.0417
TPTSR
0.3733
0.1926
0.1417
TPTSR-mirror face only
0.2800
0.1259
0.0333
TPTSR-symmetrical face only
0.3067
0.1037
0.0500
TPTSR-the proposed scheme
0.2200
0.0889
0.0250Table 4
Classification error rates of different algorithms on the FERET database.
The number of training samples
1
2
3
CRC-original
0.5567
0.4160
0.5563
CRC-mirror face
0.5550
0.4050
0.5012
CRC-symmetrical face
0.5492
0.3950
0.4975
CRC-the proposed scheme
0.5417
0.3950
0.4625
INNC
0.5567
0.4170
0.4950
INNC-mirror face
0.5550
0.4240
0.4875
INNC-symmetrical face
0.5492
0.4100
0.4900
INNC-the proposed scheme
0.5408
0.4080
0.4713
SFRFR
0.5692
0.3840
0.4625
SFRFR-mirror face only
0.5550
0.3700
0.4150
SFRFR-symmetrical face only
0.5492
0.3630
0.4138
SFRFR-the proposed scheme
0.5492
0.3400
0.3987
TPTSR
0.5133
0.3700
0.4250
TPTSR-mirror face only
0.4942
0.3430
0.3713
TPTSR-symmetrical face only
0.4842
0.3390
0.3588
TPTSR-the proposed scheme
0.4692
0.3200
0.3400Table 5
Classification error rates of different algorithms on the FLW database.
The number of training samples
1
2
3
CRC-original
0.8437
0.7660
0.7641
CRC-mirror face
0.8282
0.7485
0.7276
CRC-symmetrical face
0.8359
0.7558
0.7359
CRC-the proposed scheme
0.8178
0.7485
0.7209
INNC
0.8437
0.7718
0.7674
INNC-mirror face
0.8282
0.7747
0.7425
INNC-symmetrical face
0.8359
0.7631
0.7425
INNC-the proposed scheme
0.8178
0.7631
0.7342
SFRFR
0.8437
0.7922
0.7724
SFRFR-mirror face only
0.8282
0.7631
0.7392
SFRFR-symmetrical face only
0.8359
0.7645
0.7409
SFRFR-the proposed scheme
0.8178
0.7456
0.7209
TPTSR
0.8346
0.7442
0.7276
TPTSR-mirror face only
0.8217
0.7413
0.7010
TPTSR-symmetrical face only
0.8140
0.7311
0.7010
TPTSR-the proposed scheme
0.8127
0.7311
0.6910


### 4.3. The Weighted Fusion

In this paper, the proposed method uses original training sample to generate mirror training sample and left symmetrical virtual training sample and right symmetrical virtual training sample. Then use the original training samples, the mirror training samples, and the left and right symmetrical virtual training samples to perform representation based method, respectively, and the corresponding scores of test samplez with respect to the jth class can be denoted by s
j
1, s
j
2, s
j
3, and s
j
4. Combine all scores to conduct weighted score level fusion. For test samples, we use s
j
=
ω
1
s
j
1
+
ω
2
s
j
2
+
ω
3
s
j
3
+
ω
4
s
j
4 to calculate the ultimate score with respect to the test sample to z in the jth class. ω
1, ω
2, ω
3, and ω
4 represent the weights. Let ω
1
+
ω
2
+
ω
3
+
ω
4
=
1 and ω
1
≥
ω
2
+
ω
3
+
ω
4. Table 6 shows that different combinations of weights have different classification accuracies. However, different data sets may have very different optimal combinations of weights. Therefore, it is important to select a proper group of weights.Table 6
Classification error rates of different combinations of weights on the Yale database.
Different combinations of weights
Classification error rates
ω
1 = 1, ω
2 = 0, ω
3 = 0, ω
4 = 0 (original)
0.1083
ω
1 = 0.7, ω
2 = 0.3, ω
3 = 0, ω
4 = 0 (mirror face)
0.0333
ω
1 = 0.6, ω
2 = 0, ω
3 = 0.2, ω
4 = 0.2 (symmetrical face)
0.0500
ω
1 = 0.5, ω
2 = 0.1, ω
3 = 0.2, ω
4 = 0.2
0.0250
ω
1 = 0.5, ω
2 = 0.2, ω
3 = 0.15, ω
4 = 0.15
0.0333
ω
1 = 0.6, ω
2 = 0.1, ω
3 = 0.15, ω
4 = 0.15
0.0417
ω
1 = 0.7, ω
2 = 0.1, ω
3 = 0.1, ω
4 = 0.1
0.0500
ω
1 = 0.8, ω
2 = 0.1, ω
3 = 0.05, ω
4 = 0.05
0.0500
ω
1 = 0.9, ω
2 = 0.05, ω
3 = 0.025, ω
4 = 0.025
0.0833To get the optimal classification accuracy, we should select the optimal combination of weights in each database or even in each number of training samples. In this paper, we use cross validation to choose the best combination of weights; some recognition rates have a promotion. The cross validation is more reasonable for the proposed method because different conditions will have a different combination of weights. The parametersω
1 and ω
2 were selected from sets {
0.9,0.8,0.7,0.6,0.5
} and {
0.5,0.4,0.3,0.2,0.1,0
}, respectively. If ω
1
+
ω
2
≤
1, we let ω
3
=
ω
4
=
0.5
*
(
1
-
ω
1
-
ω
2
). Finally, we select the best combination from each combination to do face recognition. The cross validation method was also used to select the optimal parameters for RBC-mirror face, RBC-symmetrical face, and so on. We compare our method RBC-the proposed scheme with RBC, RBC-mirror face, and RBC-symmetrical face under the same conditions. In this paper, the RBC method includes CRC, INNC, SFRFR, and TPTSR. In particular, all the methods which are used to do experimental contrast should have the same training samples and test samples, as well as the same process of preprocessing.


## 4.1. Data Sets

We conduct a number of experiments on the ORL [40], Yale [41], FERET [42], and FLW [43] databases. Figure 5 shows some original training samples from the ORL face database. Figure 6 shows some original training samples from the Yale face database. Figure 7 shows some original samples from the FERET face database. Figure 8 shows some original samples from the FLW face database. The database is introduced as follows.Figure 5
Several original training samples from the ORL face database.Figure 6
Several original training samples from the Yale face database.Figure 7
Several original training samples from the FERET face database.Figure 8
Several original training samples from the LFW face database.ORL. This data set contains 400 face images from 40 subjects, each providing ten images. The ORL database includes variations in facial expression (smiling/not smiling, open/closed eyes) and facial details. Each of the face images contains 32 × 32 pixels. The first 1, 2, and 3 images per person were selected for training and the remaining for testing.Yale. This data set contains 14 subjects, and each subject has 11 images. The Yale database includes variations in facial expression and facial details. Each of the face images contains 100 × 100 pixels. The first 1, 2, and 3 images per person were selected for training and the remaining for testing.FERET. This data set contains 200 subjects, and each subject has 7 images. Each face sample contains 40 × 40 pixels. FERET mainly includes change of illumination and change of expression. The first 1, 2, and 3 images per person were selected for training and the remaining for testing.LFW. This data set contains more than 13,000 images of faces collected from the web. Each face has been labeled with the name of the people pictured. 1680 of the people pictured have two or more distinct photos in the database. In the experiments, 860 images from 86 people were chosen. Each person has 10 images. The first 1, 2, and 3 images per person were selected for training and the remaining for testing.


## 4.2. Experimental Results and Analysis

Actually, in the beginning of studying this topic, we tried to use both the original training samples and their virtual training samples to do recognition, but the results are not satisfying. The basic steps of the method which combine the original training samples and the generated virtual images into a new training sample set for face recognition (COVNFR) are showed in Figure9.Figure 9
Basic steps of the COVNFR method for face recognition.We use the collaborative representation based classification (CRC) method to do the experiment in ORL, Yale, FERET, and FLW databases. The experimental results are showed in Table1.Table 1
Classification error rates of the COVNFR method.
Database
The number of training samples
1
2
3
CRC-original
0.3194
0.1656
0.1393
ORL
CRC-virtual training samples only
0.9306
0.9313
0.9643
CRC-COVNFR
0.3778
0.2719
0.2750
CRC-original
0.3733
0.1556
0.1083
Yale
CRC-virtual training samples only
0.9000
0.9333
0.9667
CRC-COVNFR
0.3533
0.1259
0.1000
CRC-original
0.5567
0.4160
0.5563
FERET
CRC-virtual training samples only
0.9592
0.8810
0.7937
CRC-COVNFR
0.6200
0.5290
0.6525
CRC-original
0.8437
0.7660
0.7641
FLW
CRC-virtual training samples only
0.9470
0.9608
0.9269
CRC-COVNFR
0.9031
0.8314
0.8306From Table1, we find that the COVNFR method has higher classification error rates than the original face recognition methods. The reasons why the COVNFR method cannot achieve a satisfying result are the following: if we use the virtual training samples only to do face recognition, all of the classification rates in Algorithm 1 are higher than 90% which means the virtual training sample cannot inherit all of the features. However, in the condition of limited training sample, only original training samples cannot conclude comprehensive features, while the virtual training samples inherit some features from the original training samples. Therefore, the use of weight combination to combine the original training samples and their virtual training samples not only adds feature information, but also regulates the effect of each part of training samples.Our scheme is based on a weighted fusion of the original training samples, the mirror virtual faces, and the symmetrical virtual faces which can be applied to lots of face recognition methods. In this paper, we conduct contrast experiments on some representative RBC methods, which include the collaborative representation classification (CRC) method [24], an improvement to the nearest neighbor classifier (INNC) method [25], a simple and fast representation based face recognition (SFRFR) method [26], and a two-phase test sample sparse representation (TPTSR) method [27]. From Tables 2, 3, 4, and 5, we compare some RBCs with different ways to process training samples. In Algorithm 1, “CRC-original” represents the original CRC method; “CRC-mirror face” represents integrating the original face image and its mirror sample to perform CRC method; “CRC-symmetrical face” represents integrating the original face image and its symmetrical sample to perform CRC method; “CRC-the proposed scheme” represents integrating the original face image and its fused virtual sample to perform CRC method. As can be seen from these four tables, the proposed scheme performs well when compared to all the other methods. For the LFW database, there are more complex variations in the face sample than other databases. However, the proposed method still achieves a satisfactory performance.Table 2
Classification error rates of different algorithms on the ORL database.
The number of training samples
1
2
3
CRC-original
0.3194
0.1656
0.1393
CRC-mirror face
0.2944
0.1469
0.1321
CRC-symmetrical face
0.3056
0.1500
0.1286
CRC-the proposed scheme
0.2944
0.1437
0.1179
INNC
0.3194
0.2125
0.2179
INNC-mirror face
0.2944
0.2125
0.2000
INNC-symmetrical face
0.3000
0.2094
0.2000
INNC-the proposed scheme
0.2944
0.1938
0.1750
SFRFR
0.3194
0.1969
0.1750
SFRFR-mirror face only
0.2944
0.1750
0.1250
SFRFR-symmetrical face only
0.3000
0.1656
0.1179
SFRFR-the proposed scheme
0.2944
0.1656
0.0929
TPTSR
0.2861
0.1250
0.1000
TPTSR-mirror face only
0.2722
0.1219
0.0893
TPTSR-symmetrical face only
0.2806
0.1094
0.0821
TPTSR-the proposed scheme
0.2667
0.1094
0.0750Table 3
Classification error rates of different algorithms on the Yale database.
The number of training samples
1
2
3
CRC-original
0.3733
0.1556
0.1083
CRC-mirror face
0.2800
0.1259
0.0333
CRC-symmetrical face
0.3067
0.1037
0.0500
CRC-the proposed scheme
0.2200
0.0889
0.0250
INNC
0.3733
0.2296
0.1833
INNC-mirror face
0.2800
0.1852
0.1250
INNC-symmetrical face
0.3067
0.2000
0.1583
INNC-the proposed scheme
0.2200
0.1630
0.1000
SFRFR
0.3733
0.2148
0.1417
SFRFR-mirror face only
0.2800
0.2000
0.0833
SFRFR-symmetrical face only
0.3067
0.1778
0.1167
SFRFR-the proposed scheme
0.2200
0.1333
0.0417
TPTSR
0.3733
0.1926
0.1417
TPTSR-mirror face only
0.2800
0.1259
0.0333
TPTSR-symmetrical face only
0.3067
0.1037
0.0500
TPTSR-the proposed scheme
0.2200
0.0889
0.0250Table 4
Classification error rates of different algorithms on the FERET database.
The number of training samples
1
2
3
CRC-original
0.5567
0.4160
0.5563
CRC-mirror face
0.5550
0.4050
0.5012
CRC-symmetrical face
0.5492
0.3950
0.4975
CRC-the proposed scheme
0.5417
0.3950
0.4625
INNC
0.5567
0.4170
0.4950
INNC-mirror face
0.5550
0.4240
0.4875
INNC-symmetrical face
0.5492
0.4100
0.4900
INNC-the proposed scheme
0.5408
0.4080
0.4713
SFRFR
0.5692
0.3840
0.4625
SFRFR-mirror face only
0.5550
0.3700
0.4150
SFRFR-symmetrical face only
0.5492
0.3630
0.4138
SFRFR-the proposed scheme
0.5492
0.3400
0.3987
TPTSR
0.5133
0.3700
0.4250
TPTSR-mirror face only
0.4942
0.3430
0.3713
TPTSR-symmetrical face only
0.4842
0.3390
0.3588
TPTSR-the proposed scheme
0.4692
0.3200
0.3400Table 5
Classification error rates of different algorithms on the FLW database.
The number of training samples
1
2
3
CRC-original
0.8437
0.7660
0.7641
CRC-mirror face
0.8282
0.7485
0.7276
CRC-symmetrical face
0.8359
0.7558
0.7359
CRC-the proposed scheme
0.8178
0.7485
0.7209
INNC
0.8437
0.7718
0.7674
INNC-mirror face
0.8282
0.7747
0.7425
INNC-symmetrical face
0.8359
0.7631
0.7425
INNC-the proposed scheme
0.8178
0.7631
0.7342
SFRFR
0.8437
0.7922
0.7724
SFRFR-mirror face only
0.8282
0.7631
0.7392
SFRFR-symmetrical face only
0.8359
0.7645
0.7409
SFRFR-the proposed scheme
0.8178
0.7456
0.7209
TPTSR
0.8346
0.7442
0.7276
TPTSR-mirror face only
0.8217
0.7413
0.7010
TPTSR-symmetrical face only
0.8140
0.7311
0.7010
TPTSR-the proposed scheme
0.8127
0.7311
0.6910


## 4.3. The Weighted Fusion

In this paper, the proposed method uses original training sample to generate mirror training sample and left symmetrical virtual training sample and right symmetrical virtual training sample. Then use the original training samples, the mirror training samples, and the left and right symmetrical virtual training samples to perform representation based method, respectively, and the corresponding scores of test samplez with respect to the jth class can be denoted by s
j
1, s
j
2, s
j
3, and s
j
4. Combine all scores to conduct weighted score level fusion. For test samples, we use s
j
=
ω
1
s
j
1
+
ω
2
s
j
2
+
ω
3
s
j
3
+
ω
4
s
j
4 to calculate the ultimate score with respect to the test sample to z in the jth class. ω
1, ω
2, ω
3, and ω
4 represent the weights. Let ω
1
+
ω
2
+
ω
3
+
ω
4
=
1 and ω
1
≥
ω
2
+
ω
3
+
ω
4. Table 6 shows that different combinations of weights have different classification accuracies. However, different data sets may have very different optimal combinations of weights. Therefore, it is important to select a proper group of weights.Table 6
Classification error rates of different combinations of weights on the Yale database.
Different combinations of weights
Classification error rates
ω
1 = 1, ω
2 = 0, ω
3 = 0, ω
4 = 0 (original)
0.1083
ω
1 = 0.7, ω
2 = 0.3, ω
3 = 0, ω
4 = 0 (mirror face)
0.0333
ω
1 = 0.6, ω
2 = 0, ω
3 = 0.2, ω
4 = 0.2 (symmetrical face)
0.0500
ω
1 = 0.5, ω
2 = 0.1, ω
3 = 0.2, ω
4 = 0.2
0.0250
ω
1 = 0.5, ω
2 = 0.2, ω
3 = 0.15, ω
4 = 0.15
0.0333
ω
1 = 0.6, ω
2 = 0.1, ω
3 = 0.15, ω
4 = 0.15
0.0417
ω
1 = 0.7, ω
2 = 0.1, ω
3 = 0.1, ω
4 = 0.1
0.0500
ω
1 = 0.8, ω
2 = 0.1, ω
3 = 0.05, ω
4 = 0.05
0.0500
ω
1 = 0.9, ω
2 = 0.05, ω
3 = 0.025, ω
4 = 0.025
0.0833To get the optimal classification accuracy, we should select the optimal combination of weights in each database or even in each number of training samples. In this paper, we use cross validation to choose the best combination of weights; some recognition rates have a promotion. The cross validation is more reasonable for the proposed method because different conditions will have a different combination of weights. The parametersω
1 and ω
2 were selected from sets {
0.9,0.8,0.7,0.6,0.5
} and {
0.5,0.4,0.3,0.2,0.1,0
}, respectively. If ω
1
+
ω
2
≤
1, we let ω
3
=
ω
4
=
0.5
*
(
1
-
ω
1
-
ω
2
). Finally, we select the best combination from each combination to do face recognition. The cross validation method was also used to select the optimal parameters for RBC-mirror face, RBC-symmetrical face, and so on. We compare our method RBC-the proposed scheme with RBC, RBC-mirror face, and RBC-symmetrical face under the same conditions. In this paper, the RBC method includes CRC, INNC, SFRFR, and TPTSR. In particular, all the methods which are used to do experimental contrast should have the same training samples and test samples, as well as the same process of preprocessing.


## 5. Conclusion

In this paper, the proposed method integrates the original face image and its two kinds of virtual samples to perform face recognition. The scheme first exploits the original training samples to generate the mirror virtual training samples and symmetrical training samples. Then it uses the original, mirror, and symmetrical virtual training samples to represent test sample by a linear combination of the training samples. The recognition rate can be improved by adjusting the weights of original and virtual training samples based on real situations. Therefore, the proposed scheme can enhance the robustness and improve the recognition rates. The scheme performs well to overcome the problem of insufficiency of training samples. The experimental results show that the proposed method can outperform the original RBCs, the mirror virtual sample augment RBC methods, and the symmetrical virtual sample augment RBC methods.



---

*Source: 280318-2015-03-02.xml*