Proceedings of 2nd International Symposium for Color Science and Art 2020

Real-Time Emotion Estimation System
Using Face API of Microsoft Azure
Yousun Kang*, Takenori Obo*, Yuta Ogai**, and Duk Shin**Tokyo Polytechnic Univ., 1583 Iiyama, Atsugi, Kanagawa 243-0297, Japan* Department of Applied Computer Science, Facility of Engineering** Department of Electronics and Mechatronics, Facility of Engineering
2.2 Azure API (Application Programming Interface)Among various cloud computing platforms provided byMicrosoft Azure, we applied Face API to the vision part ofcognitive services. Face API can detect and compare
human faces. In addition, Face API integrates emotion
recognition, returning the confidence across a set ofemotions for each face in the image, such as anger,contempt, disgust, fear, happiness, neutral, sadness andsurprise. These emotions are understood to becommunicated universally across cultures with particularfacial expressions.

ABSTRACTThis paper presents real-time emotion estimation
system using the Face API of Microsoft Azure. The system
can display the results of estimated emotion extracted
from face image with color light. We introduce the Face
API of Microsoft Azure which is a kind of cognitive service
to identify and analyze content within images.
Experimental results showed that the proposed system is
effective for implementing an estimation of not onlyemotion but also age and gender in real-time.
1. INTRODUCTION
There will be more opportunities for communication
between human and AI (Artificial Intelligence) using
natural language. Emotional intelligence of computer
systems is increasingly important in human-AI interactions.
For example, a humanoid robot “Pepper” of SoftBank is
the personal communication robot that can read emotions,with the ability to analyze people’s expressions. Facialimage holds important quantity of attributes andinformation about a person, such as facial expression,ethnicity, gender, and age. To predict emotions from facial
expressions images, many methods are proposed inrecent years. In this paper, we propose a new system topredict emotion from facial image in real-time. The system
uses cloud computing platforms provided by MicrosoftAzure. The results of the estimated emotion display text
words in a monitor and show a color light in smart lighting
system. In the next section, we introduce the cloudcomputing and Microsoft Azure. The proposed system isexplained about configuration of hardware and software insection 3. Experimental results show in section 4. Finally,
we conclude and summarize work in section 5.

3. SYSTEM CONFIGURATION
3.1 Hardware Setup
The system comprises a web camera as an input
device, a PC based Window 10, a 60-inch monitor as adisplay, and color lights Hue in Philips as an output device.
Hue in Philips is a smart lighting system with a bridge. Thebridge acts as a smart hub, connecting with our PC tooutput color light. Fig. 1 shows the appearance of thesystem in the cololab gallery in Tokyo Polytechnic
University. To implement the system, the user has to put aswitch called the action button as shown in Fig. 2. We
prepared another monitor displaying the manual about theusage of the system with the action button.

2. FACE API OF MICROSOFT AZURE
2.1 Cloud Computing
Cloud computing platforms provide servers, storages,networking, applications over the Internet cloud with lower
cost. There are some representative services such as
AWS (Amazon Web Services), Microsoft Azure, and GCP(Google Cloud Platform). Since these commercial
services give more secure, reliable and flexible platforms
than on-premises servers, and more people are starting touse cloud computing services. The proposed systemutilizes Microsoft Azure which can operate hybrid andseamless servers integrated into edge computing
environments.

Fig. 1 The appearance of the proposed system in thegallery of cololab

9Proceedings of 2nd International Symposium for Color Science and Art 2020
3.2 Application Program
The application program is compiled by Python. By
using the specified threshold value, we can estimate anemotion among the various trained data by deep learningmodel. We can display the results to text words in themonitor and the size of the text represents the estimation
performance. Bigger text means higher performance.
When the application is executed in real-time, all results
are back up in the system.
(a) sadness(b) disgust(c) happiness(d) anger
Fig. 3 Results of estimated emotion in display monitor
Fig. 2 Display of the manual about the usage of thesystem with the action button of the other monitor
4. EXPERIMENTAL RESULTS
In this section, we show the results of the proposedsystem. Fig. 3 shows the sample images of a display foremotion estimation, age, gender. The application programis able to recognize faces from many people. Therefore,
recognized faces are simultaneously shown in the display
monitor. In addition, the smart color light system shows theresult from a person shown in Fig. 4. The smart lighting
system has the color of the rainbow such as red, orange,yellow, green, blue, indigo, and violet. Each color
corresponds with the estimated emotion as shown in
Table 1.

(a) red light

5. CONCLUSION
In this paper, we estimated emotion, age, and gender
extracted from facial image in real-time. The system candisplay the color of light according to the emotion resultswith the Face API of Microsoft Azure. The relation of colorand emotion should be investigated in psychologicaleffects for future work. Since colors and emotions areclosely linked, we can develop the communication systemusing colors psychology and smart lighting system.

(b) orange light(a) green light(b) violet light
Fig. 4 Results of various color light system according
to estimated emotion
6. REFERENCES[1] P. R. Spence, “Searching for questions, originalthoughts, or advancing theory: Human-machinecommunication”, Computers in Human Behavior, pp.
285-287, Vol. 90, (2019)[2] Microsoft Azure URL :https://docs.Microsoft.com/ja-jp/azure/cognitive-services/face/overview[3] A. S. Gremillion, “Colors and emotions: how colors
make you feel”, 99designs URL :https://en.99designs.jp/blog/tips/how-color-impacts-e
motions-and-behaviors/, (2019)

Table1 Corresponding color lights from the estimatedemotion
Colors lights
Estimated emotion
Red light
Anger
Orange light
Surprise
Yellow light
FearGreen light
Sadness
Blue light
DisgustIndigo lightContempt
Violet light
Happiness10