Face Recognition with the Multiple Constrained
Mutual Subspace Method
Masashi Nishiyama1 , Osamu Yamaguchi1 , and Kazuhiro Fukui2
1

Corporate Research & Development, Toshiba Corporation, Japan{masashi.nishiyama, osamu1.yamaguchi}@toshiba.co.jp
2
Department of Computer Science, Graduate School of Systems and Information
Engineering, University of Tsukuba, Japankfukui@cs.tsukuba.ac.jp

Abstract. In this paper, we propose a novel method named the Multiple Constrained Mutual Subspace Method which increases the accuracy offace recognition by introducing a framework provided by ensemble learning. In our method we represent the set of patterns as a low-dimensional
subspace, and calculate the similarity between an input subspace anda reference subspace, representing learnt identity. To extract eﬀective
features for identiﬁcation both subspaces are projected onto multiple
constraint subspaces. For generating constraint subspaces we apply ensemble learning algorithms, i.e. Bagging and Boosting. Through experimental results we show the eﬀectiveness of our method.

1

Introduction

Recently, many face identiﬁcation methods that perform recognition from a setof patterns instead of a single pattern have been proposed[1–5]. Since thesemethods are able to cope with variation in appearance under varying pose, arobust face identiﬁcation application can be built.
To identify faces using a set of patterns, we have previously proposed the
Mutual Subspace Method (MSM)[1]. In MSM, a set of patterns is represented
as a low-dimensional subspace. To compare the input subspace with the reference subspace representing learnt identity, we calculate their similarity which
is deﬁned by the minimum angle between the input subspace and the referencesubspace. These subspaces are generated using principal component analysis(PCA).
To improve the performance of MSM we have extended this method to theConstrained Mutual Subspace Method (CMSM)[5]. In CMSM, to extract eﬀective features for identiﬁcation, we project the input subspace and the referencesubspace onto the constraint subspace, as shown in Fig. 1. Through this projection we can extract features that are insensitive to varying facial pose and illumination, while remaining sensitive to change in individual appearance. Using
CMSM Sato et al.[6] illustrated the eﬀectiveness in a practical security system,
while Kozakaya et al.[7] demonstrated an implementation of a real-time system
on an image processing LSI chip.

2

Masashi Nishiyama et al.
Input set

Input subspace P

Reference subspace Q

PCA

Reference set

PCA
Projection

Projected input subspace P cConstraint subspace CθcProjected reference subspace Q cFig. 1. Concept of CMSM. The input subspace and the reference subspace are generated from the set of patterns. Then, both subspaces are projected onto the constraint
subspace. Finally, the similarity is determined with the angle θC .

Although CMSM is eﬀective, a large number of training patterns are requiredfor generating the constraint subspace. Since variation in appearance is large under varying pose and illumination, it is diﬃcult to acquire training patterns which
suﬃciently represent these variations. Therefore, we need a method of generating
the constraint subspace which yields high performance from a limited numberof acquired training patterns. In the ﬁeld of machine learning, ensemble learning has been proposed[8, 9]. Ensemble learning derives recognition performanceby combining hypotheses obtained from given training samples. Wang et al.[10]applied ensemble learning to face identiﬁcation based on Linear Discriminant
Analysis and demonstrated that they obtain high performance using only a few
training patterns.
In this paper we propose a new method which generates multiple constraint
subspaces by introducing the framework provided by ensemble learning. Using these constraint subspaces, we extend CMSM to the Multiple Constrained
Mutual Subspace Method (MCMSM). In MCMSM, the input subspace and thereference subspace are projected onto each constraint subspace, and the similarity is calculated on each constraint subspace. By combining these similarities we
ﬁnally determine the combined similarity as shown in Fig. 2. To generate constraint subspaces, we propose two approaches in which we apply the frameworkprovided by ensemble learning.
This paper is organized as follows. First, we describe the method for applying
MCMSM to face identiﬁcation in section 2. Next, we describe two approaches
for generating constraint subspaces in section 3. Then, we demonstrate the effectiveness of our method using MCMSM by experiments in section 4.

2

Identification using MCMSM

2.1

Algorithm for face identification

In this section, we describe the procedure of our face identiﬁcation method. First,
an input set of face patterns is obtained from a video sequence. We locate the facepattern from the positions of the pupils and the nostrils obtained automaticallyFace Recognition with the Multiple Constrained Mutual Subspace Method

P

P

Q

P c1

Q c1

θc1

...

Constraint subspace C1

3

Q
P cM

Q cM

θcM
Constraint subspace CM

Combine cos 2 θCi

Combined Similarity

Fig. 2. Concept of MCMSM. The input subspace P and the reference subspace Q are
projected onto each constraint subspace Ci. By combining M similarities(cos2 θCi ),which are calculated on Ci, we ﬁnally determine the combined similarity.

by the method described in [1, 7]. The pattern is transformed to a vector byraster-scanning of the pattern, and we apply PCA to the vectors to generate an
input subspace. Let x be a vector and NV be the number of the vectors, thebasis vectors
the input subspace are the eigenvectors of the correlation matrix
ofNV
A = 1/NV i=1
xxT [12].
To compare the input subspace with the reference subspace, registered in a
database for each individual, we calculate their combined similarity. This combined similarity is determined with similarities calculated on each constraint
subspace. The identiﬁed person is determined as corresponding to the referencesubspace of the highest combined similarity. The details of each process aredescribed in the following section.
2.2

Projection onto constraint subspaces

To project the input subspace P onto M constraint subspaces, we carry out thefollowing steps:
1. Project basis vectors of P onto the i-th constraint subspace Ci .
2. Normalize the length of each projected vector.
3. Apply Gram-Schmidt orthogonalization to the normalized vectors.
The orthogonal normalized vectors are basis vectors of the projected input subspace PCi . This procedure is repeated M times for each constraint subspace.
The projected reference subspace QCi can be obtained with the same procedure.
2.3

Calculation of the similarity on each constraint subspace

We deﬁne similarity SCi between the subspace PCi and the subspace QCi as
SCi = cos2 θCi ,(1)

where θCi represents the canonical angle between PCi and QCi . The canonicalangle is calculated using MSM[1]. The similarity SCi can be obtained from the4Masashi Nishiyama et al.

largest eigenvalue λmax of X using
Xa = λa ,X = (xmn )
xmn =

N


(2)

m, n = 1 . . . N , and(3)(ψm , φl )(φl , ψn ) ,

(4)

l=1

where ψm is the m-th basis vector of subspace PCi ; φl is the l-th basis vector ofsubspace QCi ; (ψm , φl ) is the inner product of ψm and φl ; N is the dimensionof PCi and QCi . The similarity SCi equals λmax . If the input subspace and thereference subspace are identical, the canonical angle θCi equals 0.
2.4

Combine similarities

To combine similarities obtained on each constraint subspace, we deﬁne the
combined similarity ST as follows:
ST =

M


αi SCi ,(5)

i=1

where M is the number of the constraint subspaces; αi is the i-th coeﬃcient of
Ci ; SCi is the similarity between PCi and QCi projected onto Ci .

3Generation of multiple constraint subspaces withensemble learning

In this section, we explain the method of generating a single constraint subspace for CMSM[5]. Next, we describe two approaches for generating multiple
constraint subspaces with ensemble learning for MCMSM.
3.1

Generation of a single constraint subspace

To allow for the variation in appearance for each individual, we acquire the sets ofpatterns while changing illumination and pose for L individuals. The variationof the patterns is represented as a subspace for each individual. We call thissubspace the training subspace.
To generate a constraint subspace which separates the training subspaces by
projection, we calculate eigenvectors using(P1 + P2 + . . . + PL )a = λa ,
Pj =

NB


T
ψjk ψjk
,

(6)
(7)

k=1

where Pj is the projection matrix of the j-th training subspace; NB is thedimension of training subspace; ψjk is the k-th basis vector of the j-th training
subspace. The eigenvectors, selected in ascending order, are the basis vectors of
the constraint subspace. For details of CMSM see [5, 7].

Face Recognition with the Multiple Constrained Mutual Subspace Method5

Subspace P2

Subspace P2
Subspace P3

Subspace P1

Projection

Subspace P3

Subspace P1

Projection

P3ci

P1ci

P3ci+1P1ci+1

Constraint subspace Ci
P2ci

(a)

Constraint subspace Ci+1P2ci+1(b)

Fig. 3. Concept of the method for generating constraint subspaces using Boosting.

3.2

Generation of constraint subspaces with Bagging

To generate constraint subspaces, we use Bagging[8], which is based on an ensemble learning algorithm. Multiple classiﬁers are constructed using random sampling in Bagging. To apply this framework to generating constraint subspaces, we
randomly select L (< L) subspaces from L training subspaces. Each constraint
subspace is generated independently using selected training subspaces.
Algorithm using Bagging
To summarize: we generate M constraint subspaces by the following steps:
1. Select L training subspaces randomly without replacement.
2. Generate a constraint subspace using selected L training subspaces in eq.(6).
3. Until M constraint subspaces are generated, go to 1.
3.3

Generation of constraint subspaces with Boosting

In another method of generating constraint subspaces, we use Boosting[9]. Eachclassiﬁer is constructed sequentially by reweighting the training patterns in
Boosting. The current weight is given to training patterns which were classiﬁed incorrectly in the previous constructed classiﬁer.
In applying this framework to generate constraint subspaces we must deﬁne how to calculate the weight for each training subspace. Consider similarities
between training subspaces on the constraint subspace. As shown in Fig. 3(a),when the projected training subspace P1Ci and the projected training subspace
P3Ci are similar on the constraint subspace Ci , the likelihood of the false identiﬁcation is increased for these training subspaces. To cope with this problem, we
aim to separate P1Ci+1 and P3Ci+1 on Ci+1 as shown in Fig. 3(b). To achieve this,
we generate Ci+1 by assigning large weight to P1Ci and P3Ci , thereby increasing
their importance and decreasing the remaining error.
Algorithm using Boosting
To summarize: we generate M constraint subspaces by the following steps:
1. Deﬁne the initial weight W1 (j).

6Masashi Nishiyama et al.

2. Generate the i-th constraint subspace Ci using i-th weight Wi (j) and the
projection matrix Pj of the j-th training subspace as(Wi (1)P1 + . . . + Wi (L)PL )a = λa .

(8)

3. Calculate the next weight Wi+1 (j) using Ci .
4. Until M constraint subspaces are generated, go to 2.
The weight Wi+1 (j) is calculated using
Sj
Wi+1 (j) = L
j=1 SjSj =Lβjj  ,

and

(9)

(10)

j  =jwhere βjj  equals θCijj ; θCijj is the angle between Pj and Pj  projected onto
the Ci . To generate a constraint subspace using only similar training subspaces,
we can set threshold T to becos2 θCijj T ≤ cos2 θCijj(11)βjj  =
0T > cos2 θCijj .

4

Empirical Evaluation

4.1

Performance for varying illumination

To illustrate the performance of our face identiﬁcation method, the lighting
condition was changed dynamically. We collected a video sequence at 5 frames
per second for each person under each lighting condition. We set 10 diﬀerence
lighting conditions using 7 light sources (A-G); see Fig. 4. A image of the set ofeach lighting condition is shown in Fig. 5(a). A video sequence consisted of 140
face images which were captured in arbitrary facial pose, e.g. translation, yaw
and pitch. The size of each image was 240×320 pixels and 50 diﬀerent individuals’
data were collected. From each image a 30 × 30 pixel pattern, as shown in Fig.
5(b), was extracted. This pattern was histogram equalized, resized to 15 × 15
pixels by subsampling, a vertical gradient operator was applied, and ﬁnally thepattern was transformed to a 15 × (15 − 1) = 210-dimensional vector.
We divided the data into two groups that each consisted of 25 individuals’
patterns. The ﬁrst group was used for identiﬁcation and the second for generating
constraint subspaces. In the ﬁrst group, we divided the patterns into input sets
and reference sets for each person. An input set consisted of 10 patterns foreach lighting condition and a reference set consisted of 70 patterns for eachlighting condition. We used 7 input sets per person for each lighting condition.
In the second group, to learn variation of patterns under varying illuminations, a
training subspace was generated by using all lighting condition patterns. A set of

Face Recognition with the Multiple Constrained Mutual Subspace Method(Top view)

C

Wall
470mm

400mm

800mm

C,E

G

(Side view)

D

G

7

D,F
E

CameraB

A

F

A

Lighting ConditionB

I1 I2 I3 I4 I5 I6 I7 I8 I9 I10
Cameraon : turn on the lights
blank : turn off the lights

13800mm
600mm

1700mm
1800mm
2100mm
2300mm

1200mmLight source

A on
B
on
on
on
C
D
on on
E
on
on
F
on on
G
on

Floor

Fig. 4. Setting of light sources and camera.

I1

I6

I2

I3

I4

I5

I7

I8

I9

I10

(a) captured images

I1

I6

I2

I3

I4

I5

I7

I8

I9

I10

(b) face patterns

Fig. 5. Examples of the captured images and the face patterns.

training patterns consisted of 140 patterns per lighting condition. We generated
25 training subspaces.
We compared the performance of MCMSM with those of conventional methods.
(a)Nearest Neighbor (NN)
The similarity was determined with the smallest Euclidean distance between
the pattern in the input set and the pattern in the reference set.
(b)Subspace Method[11] (SM)The similarity was determined using the average of the angle calculated between each pattern of the input set and the reference subspace. We generated
the 40-dimensional reference subspace for each reference set.
(c)Mutual Subspace Method[1] (MSM)
The similarity was determined using the angle between the input subspace
and the reference subspace. We generated the 7-dimensional input subspace foreach input set and the 7-dimensional reference subspace for each reference set.
(d)Constrained MSM[5] (CMSM)
The similarity was determined with MSM after projection onto a single constraint subspace. The constraint subspace was generated with L = 25 training
subspaces. We set the dimension of the training subspace to NB = 30, and thedimension of the constraint subspace to NC = 170.

8

Masashi Nishiyama et al.
Table 1. Experimental results under varying illumination (25 registered persons).
MethodCMR(%) EER(%)
(a)
NN
95.4
23.9
SM
95.4
12.9
(b)
MSM
95.4
9.8
(c)
CMSM
95.4
5.0
(d)
98.2
4.0
(e) MCMSM-Bagging
98.6
3.9(f) MCMSM-Boosting100
95

100 - FRR (%)

90
85
80
(a)
(b)
(c)
(d)
(e)
(f)

75
70
65
60
0

5

10

15

20

25

30

35

40

FAR (%)

Fig. 6. ROC curves (25 registered persons, varying illumination).

(e)Multiple CMSM with Bagging (MCMSM-Bagging)
The similarity was determined with MSM after projecting onto multiple constraint subspaces. Each constraint subspace was generated from L = 8 training
subspaces selected randomly. We used M = 10 constraint subspaces. The coefﬁcient of the combining was αi = 1/10.
(f )Multiple CMSM with Boosting (MCMSM-Boosting)
The similarity was determined with MSM after projecting onto multiple
constraint subspaces. Each constraint subspace was generated from weighted
training subspaces. We used M = 10 constraint subspaces. The initial weight
W1 (j)(j = 1 . . . 25) was 1/25, the threshold T was 3.5σi , and σi was the standard
deviation of similarities which were calculated between training subspaces. The
coeﬃcient αi was 1/10.
Table 1 shows the evaluation result of each method in terms of the correct
match rate (CMR) and the equal error rate (EER). CMR is the probability thatan input set of the right person is correctly accepted. EER is the probability
that the false acceptance rate (FAR) equals the false rejection rate (FRR). We
can see that the methods (e) and (f) using MCMSM are superior to (a)-(d) withregard to CMR and EER. Figure 6 shows the receiver operating characteristic(ROC) curves, which indicate FAR and FRR of each method. The superiorityof MCMSM (e) and (f) is also apparent from this.

Face Recognition with the Multiple Constrained Mutual Subspace Method
6

99

5.5

MCMSM (Bagging)

98

MCMSM (Boosting)

CMR(%)

EER(%)

9

5
4.5

97
MCMSM (Bagging)
96

4

MCMSM (Boosting)

3.5
2

4

6

8

10

Number of constraint subspaces

12

95

2

4

6

8

10

12

Number of constraint subspaces

Fig. 7. Identiﬁcation performance with increasing the number of constraint subspaces.
Table 2. Experimental results (500 registered persons).
MethodCMR (%) EER (%)
(i)
CMSM
94.7
2.3
(ii) MCMSM-Bagging
96.2
1.6(iii) MCMSM-Boosting
96.8
1.6

Figure 7 shows the performance of MCMSM versus the number of constraint
subspaces. We can see improved performance for both generating methods as
the number of constraint subspaces increased.

4.2

Performance assessment on a large database

To evaluate performance for a large number of individuals, we collected a total of1000 input sets for 500 people. The facial pose changed irregularly at each input
set although the lighting conditions were almost uniform. As before the dimension of the vector was 210. An input set consisted of 15 patterns and a reference
set consisted of 125 patterns. We compared the performance of three methods:(i)CMSM, (ii)MCMSM-Bagging and (iii)MCMSM-Boosting. In these methods,
we used the 7-dimensional input subspace and the 7-dimensional reference subspace. We used 500 training subspaces for generating the constraint subspace.
The training subspace was generated with the reference set. The dimension of
the training subspace was NB = 10, and the dimension of the constraint subspace was NC = 170. In (i), we used a single constraint subspace generated with
500 training subspaces. In (ii), we used M = 10 constraint subspaces. Each constraint subspace was generated from L = 30 training subspaces. The coeﬃcient
αi was 1/10. In (iii), we used M = 10 constraint subspaces. The initial weight
W1 (j) was 1/500, the threshold T was 5σi , and the coeﬃcient αi was 1/10.
Table 2 shows the evaluation result of each method. We can see that themethods using MCMSM are superior to that using CMSM.

10

5

Masashi Nishiyama et al.

Conclusion

This paper presented the Multiple Constrained Mutual Subspace Method in which
we applied ensemble learning to the Constrained Mutual Subspace Method. To
extract eﬀective features for face identiﬁcation, we project the input subspace
and the reference subspace onto multiple constraint subspaces. In the experiment
we obtained high performance compared with projecting onto a single constraint
subspace. To generate constraint subspaces, we apply the framework provided
by ensemble learning, i.e. Bagging, Boosting. We evaluated the algorithms ona database of varying illumination and a database with 500 individuals. The
eﬀectiveness of MCMSM is demonstrated on both databases.

References
1. Yamaguchi, O., Fukui, K., and Maeda, K.: Face recognition using temporal image
sequence. Proceedings IEEE Third International Conference on Automatic Faceand Gesture Recognition, (1998) 318-323
2. Shakhnarovich, G., Fisher, J.W., and Darrell, T.: Face Recognition from LongTerm Observations. Proceedings of European Conference on Computer Vision,(2002) 851-868
3. Wolf, L., and Shashua, A.: Learning over Sets using Kernel Principal Angles. Journal of Machine Learning Research, 4:913-931, (2003) 851-868
4. Arandjelovic, O., and Cipolla, R.: Face Recognition from Image Sets using Robust
Kernel Resistor-Average Distance. The First IEEE Workshop on Face Processingin Video, (2004)
5. Fukui, K., and Yamaguchi, O.: Face Recognition Using Multi-viewpoint Patterns
for Robot Vision. 11th International Symposium of Robotics Research, (2003) 192201
6. Sato, T., Sukegawa, H., Yokoi, K., Dobashi, H., Ogata, J., and Okazaki, A.,:
“FacePass” – Development of a Face-Recognition Security System Unaﬀected by
Entrant’s Stance. The Institute of Image Information and Television Engineers
Transactions, Vol.56, No. 7, (2002), 1111-1117, (in Japanese).
7. Kozakaya, T., and Nakai, H.: Development of a Face Recognition System on an
Image Processing LSI chip. The First IEEE Workshop on Face Processing in Video,(2004)
8. Breiman, L.: Bagging Predictors. Machine Learning, Vol. 24, No. 2, (1996) 123-140
9. Freund, Y., and Schapire, R.E.: A Decision-Theoretic Generalization of On-line
Learning and an Application to Boosting. Journal of Computer and System Sciences, Vol.55, No.1, (1997) 119-139
10. Wang, X., and Tang, X.: Random Sampling LDA for Face Recognition. Proceedings
IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, (2004) 259265
11. Watanabe, S., and Pakvasa, N.: Subspace method of pattern recognition. Proceedings of the 1st International Joint Conference on Pattern Recognition (1973) 25-32
12. Oja, E.: Subspace Methods of Pattern Recognition. Research Studies Press, England, (1983)