Who are you? – A Wearable Face Recognition System toSupport Human Memory
Yuzuko Utsumi, Yuya Kato, Kai Kunze, Masakazu Iwamura, Koichi Kise
Graduate School of Engineering, Osaka Prefecture University
1-1 Gakuencho, Naka, Sakai Osaka 599-8531, Japan{yuzuko,masa,kise}@cs.osakafu-u.ac.jp , {katou,kai}@m.cs.osakafu-u.ac.jp

ABSTRACTIn this paper, we present a wearable system for real-time
face recognition. Although there is a lot of research in facerecognition with relatively high accuracy rates, the computational time of conventional face recognition methods isin proportion to the number of faces in the database. As
the number of people in the database increases, the system
becomes difficult to recognize faces in real-time. In orderto decrease computational time and achieve real-time facerecognition, we propose to employ a coarse-to-fine recognition method and decrease the computational time even if
the number of people in the database increases. In experiments, we evaluated how the proposed system is effective
for decreasing the computational time for recognition. As
results, the proposed system can recognize faces in 238 [ms]using a database which consists of 76 people and the computational time with the coarse-to-fine recognition is 668 [ms]less than that without the coarse-to-fine recognition. Theresults support that the proposed system has enough ability
to recognize faces in real-time.
We also evaluate how our system performs in a real officeenvironment using an experiment with a database of 10 people.

Categories and Subject Descriptors
H.4 [Information Systems Applications]: Miscellaneous;
D.2.8 [Software Engineering]: Metrics—complexity measures, performance measures

Keywords
Face recognition, coarse-to-fine recognition, wearable system

1.

INTRODUCTION

Have you ever experienced that you cannot remember thename of a person you meet again? In such a situation, didn’t
you think how good it would have been if you had had a system that tell you the name of the person in secret? We pro-pose a wearable system of real-time face recognition, thatcan help you in such situations and can keep a log about
people you interacted with during the day to augment yourmemory and help you remember better. Multiple of unobtrusive systems augmenting human memory have been proposed [2], yet to our knowledge nobody implemented andexperimented with a system doing real time face recognition on a portable display. What would be the requirementsof the system? To circumvent awkward situations, first, thesystem should recognize faces unnoticed by bystanders or
the person you are meeting. Second, We need high recognition accuracy, as using the wrong name might be fatal.
The third one is computational time. According to Thrope
et. al., it takes an average of 445 [ms] for a person to recognize and react to a complex visual scene [8].This is an
important limit. To enable seamless social interaction oursystem should perform the face recognition task at least in 2x 445 [ms] (the minimum time it could take before a person
recognizes that his/her name was not remembered due tovisual clues). Additionally, the system should work for allcontacts the user might encounter at least during one day.
According to Cross et. al., we interact with around 10-12people daily (numbers based on people working and livingin the United States [3]). Unfortunately, we could not findany information on how much these numbers vary and how
difficult it is to predict which people an average user might
encounter during the day.
The first requirement could be satisfied if you always wearsuch a device and the recognition result can be reported only
to you. Our solution is a head mounted display, which canlet the user see the candidates of the person (face recognition results). The second and third requirements are closely
related to each other and there are two strategies to satisfy
them. One is to employ a fast classifier. An advantage of thisstrategy is the ease of implementation since it is already fastenough. An example is a wearable face recognition system
to assist a visually impaired person employing the Eigenface
and the LDA on a low-spec hardware[5]. The main drawback of the strategy is that fast classifiers are generally notaccurate enough, especially with a large database containing
many face images. Thus satisfying the second requirementwith a large database is not easy for a fast classifier. The
other is to employ an accurate classifier. Although its discriminant ability is satisfactory, this kind of classifiers are
generally slow and their computational time is often proportional to the number of candidates. Therefore, in order
to use such a classifier for a real-time system, a reduction

Head Mounted Display
Camera

HMD and Camera
Recognition Resulton the Head Mounted Display!

Recognition result

Fudai Taro!

Captured video
Face Detection

PC

Face Tracking

Figure 1: The proposed wearable face recognition
systemFace Recognition
CoarseRecognitionin computational time is unavoidable. In the current paper,
we take the latter strategy. We employ Affine hull based
face recognition method[1] shown to be highly accurate in
the literature. The secret to reduce the computational timeis the coarse-to-fine approach. We propose a coarse-to-finemethod suitable for the classifier, which preserves the accuracy with less computational time.

FineRecognition

Figure 2: Procedure of the proposed face recognition system

The contributions of our work are summarized as follows:
1. We discuss the design and implementation details of awearable system capable of augmenting human memory by vision-based, realtime face recognition.
2. We propose a 2 step recognition approach from coarseto-fine grain to boost the execution time towards thesocial acceptable limit of 900 [ms].
3. In experiments, we evaluate the computational timeand recognition rate. As results, the proposed system could recognize a face in 238 ms with the the cumulative recognition rate at the 10th rank was 93.3%. Computational time with the coarse-to-fine search
was 668 ms less than that without coarse-to-fine searchand the results showed that the proposed system hasenough ability to recognize faces in real time.

2.

SYSTEM OVERVIEW

A proposed wearable face recognition system consists of acamera, a head mounted display (HMD) and a computer,shown in Fig. 1. The camera is LOAS MCM-14 with aresolution of 1280 × 1024 pixels and rate of 30 frames persecond(fps). The HMD is EPSON BT-100 and the resolution of the display is 960 × 540. We mount the camera on
the HMD so it captures a scene close to the user’s field ofview. The captured video is displayed on the HMD andsent to the computer. Face detection, tracking and recognition are performed on the video and recognition results aresuperimposed on the video displayed on the HMD.
The procedure of the proposed face recognition system isshown in Fig. 2. First, the system detects faces from the captured video. Subsequently, the system tracks the detected
face, as our face recognition method requires multiple images for recognition. Thus, when the system collects a large
enough number of face images, face recognition is applied tothe collected face images. A recognition result is sent to theHMD and shown to the user. In the current implementation, face detection is executed when the face tracking fails
and the face recognition is executed when a certain numberof face images are collected. Details follow.
In order to execute face recognition in real time, we needfast face detection. In the proposed system, we employ thecascade face detector using the Adaboost algorithm by Violaet al. [9]. We use the fast face tracking method[4] to trail
the detected faces, which can track a face with up to 100
fps. Tracking results contain not only frontal faces but alsoslightly turned faces. After tracking faces, the face regions
are cropped from raw images. The system recognizes a facewith the cropped images. A feature vector is created directlyfrom pixel values of a cropped face image. For example,
when the resolution of the cropped face image is 20x20, thedimensionality of the feature vector is 400.
Recognition is divided into two parts; coarse recognition andfine recognition. In the coarse recognition, the system uses
the average of feature vectors. The system calculates the average of feature vectors for each person in the database. As
soon as there are enough images recorded for recognition,
the system calculates the average over them. The system
matches the database and the collected images by comparing the two averages of feature vectors. Then the top N ofmatching results in the coarse recognition are used for fine
face recognition. After the coarse recognition, the Affine
hull based face recognition method is applied to the top N
candidates selected in the coarse recognition process in thefine recognition [1]. Finally the top M recognition resultsin the fine recognition are displayed on the HMD to let theuser see the candidates of the person.

3. SYSTEM EVALUATION
3.1 Evaluation environments
In this section, we show the face recognition performance ofthe proposed system. We made a face database for evalu-Figure 3: Examples of database images

We stored 200 images for each person. A query sequence
consists of 16 sequential images obtained by tracking. 16
feature vectors are calculated for the 16 sequential images.
Then, they are used for recognition as a query sequence.
The number of people for queries is also 76. The PC usedfor evaluation had core i7 2600K 3.40GHz CPU and 8GBRAM. The OS of the PC was Windows 7 Enterprise 64bits.

3.215102030 40
Rank50607076

Figure 4: Cumulative recognition rates for thecoarse face recognition
were uniform. This showed that the system can recognize
faces in the same accuracy as the recognition without thecoarse-to-fine recognition when the number of people chosen
in the coarse recognition is 20. Figure 6 shows that computational time went up when the number of people chosen in
the coarse recognition. The computational time was linear
to the number of people chosen in coarse recognition. The
computational time when the number of people chosen incoarse recognition was 20 was 238 ms. The computationaltime without coarse-to-fine recognition was 906 ms. This
showed that the proposed method can recognize faces in
668 ms less than the method without coarse-to-fine recognition with keeping the recognition rate. Figure 7 shows
the cumulative recognition rates of the system when the top
20 candidates were selected in the coarse recognition, whichshows that the proposed system can show 10 candidates ofthe person with 93.3% of accuracy.
83

Coarse-to-fine face recognition

In order to evaluate the efficiency of face recognition, weexecute recognition experiments. First of all, we evaluate
the coarse recognition. We just executed the coarse recognition using the average feature vectors and calculate the top
N cumulative recognition rates. The recognition rates areshown in Fig. 4. The horizontal axis shows the rank and thevertical axis shows the cumulative recognition rates. Thegraph shows that the cumulative face recognition rate at
at the 40th rank was 100%. Next, we evaluated the recognition results and the computational time of the proposedsystem. We changed the number of people that is chosen
in coarse recognition, and calculated the recognition rates
and computational time. Recognition rates and computational time are shown in Figs. 5 and 6. In both graphs,
horizontal axes show the number of people chosen in coarse
recognition and vertical axes show the rank-1 recognition
rates and computational time for recognition. From Fig. 5,
recognition rates went up when the the number of people
chosen in coarse recognition was between 10 and 20. Whenthe number of people was more than 20, recognition rates

10095
9085
80
75
7065605550

Recognition rate [%]We applied face tracking to these videos and cropping faceregions from videos. The size of face images were normalizedto 20 × 20 [pixel] and we converted these images into 8 bits
gray scale images. Some cropped exampled images from ourdataset are shown in Fig. 3. 8 bits gray scale pixel values
are used as feature vectors.

Cumulative recognition rate [%]ating the proposed system. The face database consists ofthe VidTIMIT Audio-Video Dataset [7], the Honda/UCSD
Video Database [6] and a database from people working atour department. We merged those databases in order toincrease the number of faces for testing. We used 42 people from the VidTIMIT database consisting of color videos
of 512 × 384 pixel, and 20 from the Honda/UCSD Video
Database consisting of gray videos of 640 × 480. Our databaseconsists of color videos from 14 people taken by the same
camera as we later use for recognition. The total number ofthe people in the merged dataset is 76.

8281807978
7776
1020304050607076

Rank of coarse classifier

Figure 5: Recognition rates of the coarse-to-finerecognition method

3.3

Overall computational time of the proposedsystem

We also evaluate computational time for detection and tracking and confirm that the proposed method can recognize aface in real time. We used 43 videos from the VidTIMIT
Audio-Video Dataset and 20 videos from the Honda/UCSD
Video Database. The detection and tracking were executedwith these videos, and calculated the average time for detection and tracking. The average time for detecting a faceComputatinoal time [ms]

1000
900
800
700
600
500
400
300
200
100
0
1020304050607076

Rank of coarse classifier

Figure 6: Total computational time of the coarseand fine recognition processes

Cumulative recognition rate [%]

9694

5.

9290888684828078761510Rank

1520

Figure 7: Cumulative recognition rates for thecoarse face recognitionwas 119 ms and the average time for tracking a face on aimage was 8 ms. Capturing a image took 33 ms (30 fps).
Thus it took 33 × 16 = 528 ms to collect 16 face images
and recognize faces 238 ms at fastest with high recognition
accuracy. If the system execute face recognition, it took
119 + 528 + 238 = 885 ms to recognize a face. This showthat the proposed system is enough to achieve real-time facerecognition.

4.

for a mobile phone. The second one is the detection andtracking. In current implementation, we just detect andtrack only one face. It is not useful when the user meet morethan two people. The system should detect faces while thesystem tracks a face. The last one is the accuracy of facerecognition. In this paper, the the cumulative recognition
rate at the 10th rank was 93.3 %. We need to improve therecognition rate to put the proposed system to practical use.
A strategy is using additional information. For example, wecan use social network information for recognition. Whenyou meet an acquaintance in working time, the probability
that you meet the person’s colleagues would increase, andwhen you meet a friend at a university, the probability thatyou meet university friends and professors would increase.
If the system gets social network information from social
networking sight like Facebook and applies the information
to the face recognition, the recognition rate will increase.

CONCLUSION

This paper presented a fast wearable face recognition system
by using coarse-to-fine recognition. In experimental results,
the computational time to show the 10 candidates of a person in front of the user was 668 ms and the accuracy was93.3%. Thus the proposed system is effective. The averagetime for a human to process and react to a complex visual
stimulus is around 500ms according to Thorpe [8], yet withhigh individual variations. Our system is very close to thismean value.
The future work is as follows. First one is that the systemdoes not have any mobility because we used a desktop PC.
We are planning to put the proposed system to a mobilephone. We should build a server-client system because thecomputational cost of detection and recognition is too muchREFERENCES[1] Hakan Cevikalp and Bill Triggs. Face recognition basedon image sets. In Proceedings of 2010 IEEE Conference
on Computer Vision and Pattern Recognition, pages
2567–2573, 2010.
[2] Vicka R Corey. The memory glasses: subliminal vs.
overt memory support with imperfect information. In
Proceedings of the Seventh IEEE International
Symposium on Wearable Computers (ISWCÕ03),
volume 1530, pages 17–00.
[3] Robert L Cross and Andrew Parker. The hidden powerof social networks: Understanding how work really getsdone in organizations. Harvard Business Press, 2004.
[4] Joao F. Henriques, Rui Caseiro, Pedro Martins, andJorge Batista. Exploiting the circulant structure oftracking-by-detection with kernels. In Proceedings ofEuropean Conference on Computer Vision, pages
702–715, 2012.
[5] Sreekar Krishna, Greg Little, John Black, andSethuraman Panchanathan. A wearable face recognition
system for individuals with visual impairments. In
Proceedings of the 7th international ACM SIGACCESS
conference on Computers and accessibility, pages
106–113, 2005.
[6] Kuang-Chih Lee, Jeffrey Ho, Ming-Hsuan Yang, andDavid Kriegman. Visual tracking and recognition using
probabilistic appearance manifolds. In Computer
Vision and Image Understanding, volume 99, pages
303—331, 2005.
[7] C. Sanderson and B. C. Lovell. Multi-region
probabilistic histograms for robust and scalable identity
inference. In Lecture Notes in Computer Science(LNCS), volume 5558, pages 199–208, 2009.
[8] Simon Thorpe, Denis Fize, Catherine Marlot, et al.
Speed of processing in the human visual system. nature,381(6582):520–522, 1996.
[9] Paul Viola and Michel J. Jones. Robust real-time facedetection. International journal of computer vision,
57(2):137–154, May 2004.