Developing Real-Time Face Identification
Device Composable with Distributed
Applications
Kosuke Hirayama1(B) , Sachio Saiki1 , and Masahide Nakamura1,2
1

Graduate School of System Informatics, Kobe University,
1-1 Rokkodai, Nada, Kobe, Japanhirayama@ws.cs.kobe-u.ac.jp, sachio@carp.kobe-u.ac.jp,
masa-n@cs.kobe-u.ac.jp
2
Riken AIP, 1-4-1 Nihon-bashi, Chuo-ku, Tokyo 103-0027, Japan
Abstract. In recent years, many applications which using face identification are developed. However, development load in creation of an
application program using the face identification is high because of thetight coupling of each functions and lack of sharability of imaging device.
To address this problem, in this research, we propose a new device
named “face identification sensor box” which can realize loose couplingof function and high sharability. The proposed device uses cognitive API
for face identification and notifies the result to application with Publish/Subscribe messaging model. Using this architecture, face identification results can be shared among with applications and design of application becomes easier.
Keywords: Face identification · Cognitive API · Pub/Sub · IoT ·Edge computing

1

Introduction

For the rapid improvement of image processing and AI technology, many applications start to incorporate face identify function, which identiﬁes a person fromimages obtained from image sensors, by using computer vision. Face identiﬁcation
is a method of ﬁnding and extracting a face and calculating positions of face parts,
which can be a characteristic part when identifying a person, like eyes or a nosefrom extracted faces as feature points, and identifying persons by comparing feature points with that of other people registered in advance. In our life, face identiﬁcation is often used for user authentication, for example, iPhone X, a smartphonereleased by Apple Inc., has Face ID function which uses face identiﬁcation to unlock
their lock and complete online payment [4]. Furthermore, Universal Studio Japan,
a theme park in Japan, have incorporated face identiﬁcation in the identity conﬁrmation process which is needed when annual pass holders enter the park to improve
work eﬃciency related to identity conﬁrmations and prevent abuses [7]. In addition to user authentication, various usages are emerging. For example, developing ac Springer Nature Switzerland AG 2019

V. G. Duﬀy (Ed.): HCII 2019, LNCS 11581, pp. 420–432, 2019.
https://doi.org/10.1007/978-3-030-22216-1_31

Developing Real-Time Face Identification Device Composable421

personal talking robot which is capable of comprehending the target user and generate proper subjects for them by providing face identiﬁcation on the robot [3], andmaking a system which can prevent dementia patients’ going out and wandering
without permission [5], and so on.
Face identify function oﬀers novel added values to applications, but on the other
hand, when implementing these function to applications, some problems such as
high development load due to low reusability of the function, and lack of sharabilityof resources such as cameras and devices, often occur. To address these problems, inthis research, we propose a new device named Face identification sensor box which
can oﬀer a face identiﬁcation service which various applications can use. The sensor
boxes keep taking photos by a connected camera and trying to detect faces. Then,
when a face is detected, the sensor box does face identiﬁcation and identiﬁes whose
face is this. The face identiﬁcation process is executed by using external cognitive
service. After the face identiﬁcation completed, the sensor boxes notify the resultof the identiﬁcation to subscribed external applications with Pub/Sub messaging
model. By using the Face identiﬁcation sensor box, multiple applications that needface identiﬁcation in the same space can use the results at their desired timings byusing a single sensor box. Also, since we can delegate face identiﬁcation process tothe sensor box, developers can reduce the working load in designing applications,and they may be able to focus on the essence of the services they want to oﬀer.
Moreover, we can reduce dependency within the application and expect improvement of scalability. As an example of a set of service which can be realized by using
the Face identiﬁcation sensor box, for instance, in an oﬃce, put the device on an
entrance, and identify faces when someone pass there, then create attendance management and entering or leaving room record, after that, unlock the computer ofthe corresponding user’s and give information about their schedule and reminder.
In this paper, to realize the Face identiﬁcation sensor box, we design a functional
requirement and conﬁrm the eﬀectiveness of the proposed method by a prototype
implementation. We have checked that we can oﬀer face identiﬁcation service toexternal applications which require these service by using the prototype.

2

Preliminaries

2.1

Pub/Sub Type Messaging

Pub/Sub (Publish/Subscribe) [9] is one of the messaging models used for asynchronous exchanging messages between multiple devices and services. We show aconceptual diagram of Pub/Sub in Fig. 1. In Pub/Sub messaging model, senderclients creating messages and sending them are called Publisher, and recipientclients receiving messages are called Subscriber. Messages sent by Publishers are
stored in the destinations called Topic. On the other hand, Subscribers register(subscribes) to topics in which necessary messages are sent. When a message
stored in a topic, this message is sent (delivered) to Subscribers of this topic.
By using this messaging model, both Publishers and Subscribers can send andreceive messages with no dependency on each other’s state. Basically, Publishers

422

K. Hirayama et al.

Fig. 1. A conceptual diagram of Pub/Sub

can send messages sequentially regardless of Subscribers’ status, and Subscriber
can receive only necessary messages. As a result, the performance and scalabilityof applications can be improved.
2.2

MQTT

MQTT (Message Queue Telemetry Transport) [6] is a protocol to realize
asynchronous communication between two endpoints with Pub/Sub messaging
model. Since MQTT is an asynchronous communication protocol, MQTT communication can be stable regardless of network reliability, and since MQTT isa lightweight protocol, many IoT (Internet of Things) devices with not highperformance use MQTT.
MQTT deﬁnes two entities in the network: Broker and Client. Client corresponds to Publishers and Subscribers in Pub/Sub. Broker works as a Client’s
mediator and manages the destinations of messages for each of topics. In actual
connection, Publishers send messages to Subscribers through Broker. Whensending messages, Publishers set a topic of each message. Broker sends the message to Subscribers only which subscribe to the message’s topic. In this way,
MQTT makes able to Pub/Sub type messaging.
2.3

Cognitive API

With the rapid technological growth of cloud computing, now we can use cloudservices to process various tasks which are previously executed on edge sides.
Under such a kind of background, various services utilizing artiﬁcial intelligence,
which becomes explosively popular in recent years, are now getting available asa cloud service. One of the actual cases of cloud service using artiﬁcial intelligence is cognitive services. Cognitive services extract useful information fromvisual/sound/linguistic/knowledge data by analyzing them and enable computers to recognize these types of information. Using cognitive services can realizefunctions about cognition which are previously diﬃcult for computers. The cognitive service based on images can extract various information from images, anda lot of public clouds provide this kind of services. In general, cloud services

Developing Real-Time Face Identification Device Composable423

are available from outside as APIs. Likewise, users of cognitive services can getvarious information by sending images to the API. With cognitive services, forexample, it is able to analyze human face images and estimate the age, gender,and emotional value of the face. It is also possible to recognize various items suchas appliances, furniture, foods, and colors from images. Furthermore, It is able
to output tags about the place and weather from the background of an image.

3

Proposed Method

In this chapter, we provide a detailed explanation for the face identiﬁcation
device mentioned in Sect. 1, which various applications can use.
3.1

Problems with Existing Face Identification Application

Face identiﬁcation function oﬀers novel added values to applications. Currently,
many companies such as NEC [8] and Panasonic [2] are oﬀering face identiﬁcation
solutions. However, when implementing these function to applications, developers have to implement services, photo taking and face identiﬁcation functions foreach application. Therefore, each function has tight coupling with applications,and as a result, the reusability of these function may deteriorate. For example,
we cannot reuse photo taking and face identiﬁcation functions although these
are a general function. In addition, Occupying one camera by one application
makes another problem. In the case where multiple face identiﬁcation functions
required within the same space, although the object person is in the same place,
many cameras will be required.
3.2

Face Identification Sensor Box

Based on the problems of existing applications, in this research, we propose
Face identiﬁcation sensor box. The Face identiﬁcation sensor box is a device toprovide some functions to share cameras and face identiﬁcation services. The
sensor box has to be able to provide the same face identiﬁcation function tomultiple applications. In addition, considering when there are multiple sensor
boxes, it is required that we can conﬁgure the face identiﬁcation and add ordelete users for each device all at once. Then, we make the sensor boxes delegate
face identiﬁcation processes to an outside service, and execute and manage theface identiﬁcation function in there, to reduce the load of managing sensor boxes.
The sensor box also has to be able to deal with many pollings at the sametiming, assuming the sensor box is called by multiple applications. And in orderto improve the scalability, the sensor box should be able to continue to notify the
results of identiﬁcation. Then, we adopt a connection framework which realizes
asynchronous communication.
We show a conceptual diagram of the whole architecture of the sensor box
in Fig. 2. “Face identiﬁcation process” is the component which represents APIs
for face identiﬁcation process with the sensor box, such as Pub/Sub connection

424

K. Hirayama et al.

Fig. 2. A conceptual diagram of the whole architecture(described later) or using a cognitive service. “External server” shown on the sensor box is for building the database which uniquely stores user information andso on separately from the cognitive service. “DB accessor” is used for exchanging
data between the database on the external server and external applications.
3.3

Functional Requirement

The face identiﬁcation with the sensor box has these function:
Service Exposure with Pub/Sub. When applications need face identiﬁcation
function, applications have to register itself to the sensor box. The sensor box
stores subscribed applications. After the face identiﬁcation process, when thesensor box obtains the result, the sensor box notiﬁes the result to all subscribed
applications. After that, applications can process their tasks with received result.
We adopt Pub/Sub in the connection between the sensor box and applications
and make it asynchronous.
In addition to the function to subscribe, we equip the sensor box with afunction to unsubscribe for when applications no longer need the results.
Photo Taking and Face Detection at Fixed Intervals. The sensor box
keeps getting images from the connected camera. Before the face identiﬁcation,

Developing Real-Time Face Identification Device Composable425

as a preparation, the sensor box executes face detection locally because it is inefﬁcient to execute face identiﬁcation for all obtained images. In the face detection
process, whether an image contains a face is only judged. Thereby, the sensor
box can execute face identiﬁcation only for images containing a face.
Face Identification with Cognitive API. The sensor box executes face identiﬁcation with face images obtained on the previous section. We use a cognitive
API for the face identiﬁcation to delegate face identiﬁcation processes to theoutside. When the face identiﬁcation process ﬁnished, the sensor box receives
the result.
By the way, to execute face identiﬁcation with cognitive API, we have toregister the name and face images of object people and train the machine learning model used for identiﬁcation beforehand. We can do these operations byaccessing endpoints of cognitive API, but, to make applications can do it easily,
we have to create wrapper APIs on the sensor box. Also, when we add a newuser on cognitive API, some problem may occur on the API side, for example,
the API does not accept non-latin characters for user’s name, or the API cannot register a detailed proﬁle of users such as a birthday. Therefore, we build
a database with a unique data format on an external server. When we add a
new user of identiﬁcation, the sensor box sends only the necessary information
to the cognitive API and registers to the API, and, at the same time, sends full
information to the external server’s database and make it stored also in there.
Furthermore, in order to enable applications to use the information stored in
the database, we have to prepare ways to refer the database via the sensor box
as APIs. To prepare such necessities, we make DB accessor which is a group ofAPIs to use the database. In addition to user information, we put various setting
information such as API key and endpoint to use a cognitive API in the external
server and make the external server provide this information as a service master, to enhance the convenience when accessing from each sensor box. All user
information is stored in Face API and the external server. As a result, the sensor
box itself does not store any user information.
The above is the functions necessary for the Face identiﬁcation sensor box.

4

Implementing the Prototype

4.1

Implementation

Based on the proposed method, we implement a prototype of the Face identiﬁcation sensor box.
We use the following technologies.
– Development languages: Python 3.5, HTML5, JavaScript, PHP 5.3– Database: MySQL 5.7

426

K. Hirayama et al.

Fig. 3. The overall implementation– Libraries: Paho MQTT1 , Flask2 , Flask-SocketIO3 , OpenCV4– MQTT broker: Mosquitto5– Cognitive API: Microsoft Face API [1]Paho MQTT, Mosquitto is an open source implementation of MQTT protocol.
Paho MQTT serves as Publisher and Subscriber, and Mosquitto serves as Broker. Flask is one of the web application frameworks. We use Flask to make each1

https://www.eclipse.org/paho/.
http://flask.pocoo.org/.
3
https://flask-socketio.readthedocs.io/en/latest/.
4
https://opencv.org/.
5
https://mosquitto.org/.
2

Developing Real-Time Face Identification Device Composable

427

function WebAPIs. Flask-SocketIO is a library which enables WebSocket connection with Socket.IO on Flask applications. We use Flask-SocketIO to connect
with a browser. We use Raspberry Pi 36 (OS : Raspbian 9.4), which is a small
single board computer, as the prototype edge device.
4.2

Notification Flow of Identification Results

For the prototype, we make the part which notiﬁes results of face identiﬁcation
Pub side, and we make external applications Sub side, then Pub/Sub communication is performed between both side. Inside of the sensor box, the Face
identiﬁcation controller notiﬁes the results to Mosquitto, and Mosquitto assigns
them to the speciﬁc topic, and the Result notiﬁcation controller which subscribes
this topic receives the results of face identiﬁcation asynchronously. The Result
notiﬁcation controller works as Publisher to external applications and notiﬁes
the identiﬁcation results to URLs of subscribed external applications. By theseprocess, the Result notiﬁcation controller works as Broker simulatively.
Actually, the Face identiﬁcation sensor box also supports Pub/Sub by using
MQTT. Mosquitto can be Publisher and send the identiﬁcation results to external applications. However, we do not assume that MQTT used from applications
directly for this time, and we do not use this function.
4.3

Preparation

For the implementation of the prototype, as cognitive API, we use Microsoft
Face API (hereinafter called “Face API”) which is one of the services of Microsoft
Azure. Face API can detect faces, identify faces, analyze emotions, and so on. We
can use these functions easily via the cloud. To use face identiﬁcation functionof Face API, ﬁrstly we have to make a group (=PersonGroup) and add users(=Person) who are object people of identiﬁcation. On the registration, an ID(=personId) used by Face API is assigned to each user. By selecting the Person
registered, uploading one’s face images, training of the learning model of the
PersonGroup enables face identiﬁcation.
We can complete these operations by sending requests including various
parameters to the correspondent Face API endpoints. However, to simplify this,we make the Face API controller which is a set of utilities including various
operations to Face API.
And we build a database to store users’ information and various setting
values on an external MySQL server, and we make PHP APIs on the server sothat the sensor box can access the database. The database keeps informationof users who use face identiﬁcation in the form [uId, personId, name, reading,birthday, gender, nickname, description]. uId is the unique Id used in the Face
identiﬁcation sensor box, and personId is the Id used in Face API. We makeapplications which subscribe to the sensor box to use uId mainly. Regarding the6

https://www.raspberrypi.org/.

428

K. Hirayama et al.

use of the database, to simplify it, we make the DB controller which is a set ofutilities to get, write, modify, delete, search data and convert id mutually.
Furthermore, with the Face API controller and the DB controller, we implement the Data accessor as a WebAPI to use these controllers and use functions
to register and delete users and so on from external applications. In addition
to user registration and deletion, the Data accessor has some functions such as
to convert between uId and personId, and to conﬁrm and acquire the existenceof user information corresponding to each Id, and applications can get information from the database as needed. The reason why we avoid making eachcontroller API directly is that, when registering new entries, after registering
to Face API, we have to store it also in the database, but by gathering these
operations together with the controllers, we can register necessary informationcollectively by POST. As a result, we can simplify the implementation of applications. Also, we can prevent inconsistency between registered information ofFace API and of the database.
We show the overall implementation of face identiﬁcation sensor box
described above in Fig. 3.
4.4

Details About the Operation

In this section, we explain the functions of the prototype while associating functional requirements mentioned in Sect. 3. Note that we use URL call for Pub/Sub
connection of the prototype.
Service Exposure with Pub/Sub. Applications which need to use the Face
identiﬁcation sensor box have to register their destination URL to the endpointof the Result notiﬁcation controller. To complete this operation, applications
have to connect to the endpoint such as (DeviceURL):(PORT):/reg app url
with POST request. After the registration, when face identiﬁcation is executed,
the sensor box notiﬁes the result to all registered URLs. Applications can cancel
their registration in the same way at any time.
Photo Taking at Fixed Intervals and Face Detection. The sensor boxes
keep getting images by a connected camera and detecting a face. For face detection, we use OpenCV which is an image processing library and use Haar Cascade[10] classiﬁer with preliminarily prepared classiﬁer ﬁle. When a face is detectedin the image, the face identiﬁcation process starts.
In the face detection, the sensor box does not execute face identiﬁcation
immediately when a face is detected. Instead, we introduce Face counter tostable face identiﬁcation. We show the role and behavior of Face counter in
Fig. 4. The Face counter increases when a face is detected in the image, andthe sensor box considers the face detection completed only after Face counter
exceeds a certain threshold. Once the face is detected, the Face counter decreases
while the face detection does not succeed, and until the value of Face counter
becomes 0, the sensor box does not detect the next face. With the Face counter,

Developing Real-Time Face Identification Device Composable

429

Fig. 4. The role and behavior of Face counterthe sensor box can get face images when users keep still in front of the camera,
and the face identiﬁcation becomes stable. Besides, since Face API (describedlater) has restrictions on the number and interval of calls, it is impossible toexecute massive face identiﬁcation in a short time, but by using the Face counter,
we can reduce the number of face identiﬁcations and therefore we can deal withthis problem.
Face Identification with Cognitive API. The Face identiﬁcation controller
sends images which are obtained by the face detection to the Face API controller.
The Face API controller executes some processes to use Face API, and receive
the personId of the person with the highest conﬁdence.
Before identify faces using Face API, we have to register users. We register
users through the Data accessor. The Data accessor has an endpoint for user
registration in form like (DeviceURL):(PORT):/register and by sending necessary information there, we can register new users. The Data accessor also has
endpoints to add and delete face images, and these functions are callable byexternal accesses. The sensor box only notiﬁes uId to applications as a result of
the face identiﬁcation, and does not notify detailed information such as name.
Therefore, when applications need detailed information about users, applications
have to use the Data accessor to obtain these information using uId as the key. If
Face API received a face image of a not registered person and failed to identify,
the sensor box notiﬁes “unknown” as a result.

430

K. Hirayama et al.

Fig. 5. A screenshot of the sample program

4.5

A Sample Application and Testing

For samples using functions of the prototype of the Face identiﬁcation sensor box,
we made two sample web applications for clients, one is for user registration, andthe other is to receive the result of face identiﬁcation and display the name of theuser. These applications access to functions of the sensor box from the outside.
We tested whether the sensor box works properly by using these applications.
Firstly, we use the user registration application and register some users on abrowser. This application sends information which ﬁlled in a certain form to the
Data accessor. The Data accessor stores the information to the database, andregisters it to Face API. Then, Face API issues personId. personId is represented
in hexadecimal form like
c0a094ab-e8b0-4368-...

After that, a user takes some face photos on the browser, then, this application
sends the photos to Face API through the Data accessor.
Next, we start the results receiver application and send its URL to the Result
notiﬁcation controller to subscribe the sensor box. If the subscribe process succeeds, the sensor box returns 200 HTTP status code.
After the above preparations are completed, the user turns one’s face to thecamera connected to the sensor box, then the sensor box executes face identiﬁcation. When the face identiﬁcation completed, Face API returns personId as
result. The sensor box converts the personId to uId and sends it to subscribed
URLs. The result receiver application receives the result and sends it to the
Data accessor to get various information from uId. The Data accessor has anDeveloping Real-Time Face Identification Device Composable

431

API which returns various information of a user registered in JSON format byusing uId as a key. By sending a uId to URL /get data by uid by POST, theData accessor returns the JSON including information of the user registered inadvance. The JSON is like

Finally, this application extracts the ‘name’ attribute from the JSON anddisplays (It means “Hello, Kosuke Hirayama !” in Japanese.)

We show the screenshot of this application in Fig. 5.

5Conclusion

In this paper, we proposed the Face identiﬁcation sensor box which can realize development eﬃciency, eﬀective use of resources and so on in developing
applications which has face identiﬁcation function. In the proposed method, byusing the cognitive API of cloud service for face identiﬁcation, we can easily
realize high-quality face identiﬁcation function and manage it. Also, by incorporating Pub/Sub for communication with applications, the sensor box can asynchronously send the identiﬁcation results and secure scalability capable of bearing uses from many applications. The future tasks are not only to incorporate
this system we implemented into a large scale external application and evaluate its performance, but also to expand the sensor box to cognitive sensor box
which can realize various cognitive service other than face identiﬁcation. In addition, considering the security risk, we will have to put an authentication process
before accessing functions of the sensor box from outside, and, to prevent abusesin face identiﬁcation, we will also have to make the method by which the sensor
box can combine authentication methods other than face identiﬁcation, such as
password, with user authentication process.
Acknowledgments. This research was partially supported by the Japan Ministryof Education, Science, Sports, and Culture [Grant-in-Aid for Scientific Research (B)(16H02908, 18H03242, 18H03342), Grant-in-Aid for Scientific Research (A) (17H00731)],
and Tateishi Science and Technology Foundation (C) (No.2177004).

432

K. Hirayama et al.

References
1. Face API - Facial recognition software – Microsoft Azure. https://azure.microsoft.
com/en-us/services/cognitive-services/face/
2. Faceproface recognitionpanasonic security system. https://security.panasonic.
com/Face Recognition/
3. Facial recognition: coming to a gadget near you. https://techxplore.com/news/
2019-01-facial-recognition-gadget.html
4. iphone xr - face id - apple. https://www.apple.com/iphone-xr/face-id/
5. A measure to use face identification system to prevent going out from medical/care facilities without permission – face identification based wandering prevention system for dementia patients [lykaon]. https://www.facial-lykaon.com/topics/
wandering prevention.php (in Japanese)
6. MQTT. http://mqtt.org/
7. New biometric identification tools used in theme parks — NEC. https://www.nec.
com/en/global/about/mitatv/03/2.html
8. Public safety: Products & solutions — NEC. https://www.nec.com/en/global/
solutions/safety/Technology/FaceRecognition/index.html
9. Birman, K., Joseph, T.: Exploiting virtual synchrony in distributed systems.
SIGOPS Oper. Syst. Rev. 21(5), 123–138 (1987)
10. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple
features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2001, vol. 1, p. I, December 2001