39
Evaluation of Facial Expression Recognition by A Smart Eyewear for
Facial Direction Changes, Repeatability and Positional Drift
KATSUTOSHI MASAI, Keio University
MASA OGATA, National Institute of Advanced Industrial Science Technology
YUTA SUGIURA, Keio University
KAI KUNZE, Keio University
MASAHIKO INAMI, The University of Tokyo
MAKI SUGIMOTO, Keio University

This paper presents a novel smart eyewear that recognizes the wearer’s facial expressions in daily scenarios.
Our device uses embedded photo reflective sensors and machine learning to recognize the wearer’s facial
expressions. Our approach focuses on skin deformations around the eyes that occur when the wearer changes
their facial expressions. With small photo-reflective sensors, we measure the distances between the skinsurface on the face and the 17 sensors embedded in the eyewear frame. A Support Vector Machine (SVM)
algorithm is then applied to the information collected by the sensors. The sensors can cover various facialmuscle movements. In addition, they are small and light enough to be integrated into daily-use glasses. Ourevaluation of the device shows the robustness to the noises from the wearer’s facial direction changes andthe slight changes in the glasses’ position, as well as the reliability of the device’s recognition capacity.
The main contributions of our work are as follows: (1) We evaluated the recognition accuracy in daily scenes,
showing 92.8% accuracy regardless of facial direction and removal/remount. Our device can recognize facial
expressions with 78.1% accuracy for repeatability and 87.7% accuracy in case of its positional drift. (2) We
designed and implemented the device by taking usability and social acceptability into account. The device
looks like a conventional eyewear so that users can wear it anytime, anywhere. (3) Initial field trials in adaily life setting were undertaken to test the usability of the device.
Our work is one of the first attempts to recognize and evaluate a variety of facial expressions with anunobtrusive wearable device.
Additional Key Words and Phrases: Wearable, Facial expression, Affective Computing, Eyewear Computing
ACM Reference Format:
Katsutoshi Masai, Yuta Sugiura, Masa Ogata, Kai Kunze, Masahiko Inami, Maki Sugimoto, 2016. Smart
Eyewear that Recognizes A Wearer’s Various Facial Expression in Daily Life ACM Trans. Interact. Intell.
Syst. 9, 4, Article 39 (March 2017), 23 pages.
DOI: http://dx.doi.org/10.1145/3012941

1. INTRODUCTION

Sometimes we find ourselves among others who speak a language that we do not understand. Even in that situation, however, it is still possible to guess some contextual
information by observing nonverbal clues: e.g. if they are fighting or in love, or if it
is a business or private meeting. Even when we communicate in the same language,
we rely heavily on nonverbal clues such as pauses, gestures and the tone of the voice.
Some estimate that 60 to 90% of our daily communication is nonverbal [Mehrabian
1972; Knapp et al. 2013]. While these nonverbal clues vary in kind (some auditory andsome visual), facial expressions are perhaps among the most accessible and important.
This work is supported by JST CREST.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted
without fee provided that copies are not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights for components of this work owned
by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request
permissions from permissions@acm.org.
c 2017 ACM. 2160-6455/2017/03-ART39 $15.00
DOI: http://dx.doi.org/10.1145/3012941
ACM Transactions on Interactive Intelligent Systems, Vol. 9, No. 4, Article 39, Pub. date: March 2017.

39:2

K. Masai et al.

Fig. 1. User wears our smart eyewear. It canrecognize wearer’s facial expressions.

Fig. 2. Our Smart eyewear recognizes eight universal facial expressions

As nonverbal clues play an essential role in our everyday interpersonal interactions,
it seems natural to incorporate them in the field of Human-Computer Interaction.
As computing systems become increasingly ubiquitous and support us in everyday
situations, they need to be able to process more contextual information to improve thequality of human-computer interactions. Recognizing facial expressions would be an
important step towards improving user experience.
Facial expressions are vital in communicating a person’s intentions and emotional
states. Affective Computing explores the possibility of incorporating human affection
in computing [Picard 1995; Calvo and D’Mello 2010]. A device that recognizes facial
expressions may open up new opportunities for more naturalistic user experience inhuman-computer interactions since facial expressions provide rich information about
our emotional states [Keltner et al. 2003]. A common approach to recognizing facial
expressions is computer vision based, which analyzes recorded images to identify certain expressions. However, this method is only reliable in an experimental setting, andthere is still relatively little research that tries to detect facial expressions in a daily
life scenario.
Our goal is to create an unobtrusive and truly wearable device that recognizes people’s facial expressions in everyday life. In order to achieve this goal, we have developed a device that can recognize facial expressions, specifically in the form of a smart
eyewear (see Figure 1). We believe that ”wearability” is important for tracking users’emotional states for a long term. In this project, we focus on skin deformations aroundthe eyes caused by the movement of facial muscles in order to detect facial expressions
in an efficient and minimally obtrusive way. We use 17 photo reflective sensors that are
integrated into the front frame of the glasses to detect the skin deformation around theeyes. The sensors we used for our prototype are small enough that they can be potentially integrated into glasses for everyday usage. Our approach improves the usability
in a daily life setting, compared to the camera-based systems. Our device is also ”wearable” in terms of social acceptability as the design follows that of conventional eyewear.
The high wearability can, in turn, translate into higher tracking ability. It is the kindof wearable device that can provide a good vehicle for understanding the user’s affective patterns in day-to-day scenarios, as discussed by Picard et al. [Picard and Healey
1997]. Our method can detect various facial movements robustly by applying SVM tothe data collected from the 17 photo reflective sensors that are embedded in the front
frame of the glasses.
ACM Transactions on Interactive Intelligent Systems, Vol. 9, No. 4, Article 39, Pub. date: March 2017.

Evaluation of Facial Expression Recognition by A Smart Eyewear

39:3

In this paper, we focus on recognizing eight universal facial expressions (see Figure 2). In addition to the universal six facial expressions (happiness, disgust, anger,
surprise, fear, and sadness) defined by Ekman [Ekman 1989], we include contempt,
which is sometimes considered as a universal facial expression [Matsumoto 1992] and”neutral” as a baseline for detection. We will discuss the possible usage scenarios andoffer some insights into our affective states acquired from our system. More complex
facial expressions such as a fake smile or a ”happy surprise” are out of the scope of ourcurrent project, though they will be interesting signals to tackle in the future.
The aims of our work are as follows: (1) Recognizing the user’s expressions in theeight universal categories (neutral, happiness, disgust, anger, surprise, fear, sadness,and contempt) with a wearable device in a reliable and robust manner. We used 17
photo reflective sensors that cover most of the facial muscles movements. The acquired
data are applied to SVM for robust recognition. We evaluated the robustness in daily
scenarios: when the user changes the head direction; when the user uses the device
on different days; when the user walks; and when the device slips down the nose. (2)Designing a truly wearable device that is socially acceptable and can be used anytime,
anywhere. We designed and implemented our system in the form of a fully-packaged,
conventional-looking eyewear, so the device comfortably fits the context of daily usage.
To this end, we used small photo reflective sensors that are small and light enough to
be integrated into everyday glasses. (3) Observing and analyzing the long-term distribution of the user’s facial expressions. This work is an extended version of the previous
submission [Masai et al. 2016].
2. RELATED WORK
2.1. Facial Expression Recognition With A Camera

There are some notable works related to facial expression recognition in the fieldof computer vision. The two main streams are the recognition of affect and therecognition of facial action units [Zeng et al. 2009]. According to Tian et al., thegeneral approach to automatic facial expression analysis (AFEA) consists of threesteps: Face acquisition, facial data extraction and representation, and facial expression recognition [Tian et al. 2011]. One AFEA approach achieved an accuracy of
91.5% for recognizing basic expressions [Littlewort et al. 2002] using the Cohn-Kanade
Database [Kanade et al. 2000]. However, there are three major problems with camerabased systems regarding usability in a daily life setting. First is the limited field ofview for tracking. Most of the previous works used a single fixed camera installed in
the environment. As the position of the camera is fixed, it has difficulty in tracking
the users continuously over a wide area for a long period. Tracking the users becomes
especially difficult when they move constantly, or there is an obstacle between themand the camera. Secondly, the AFEA camera systems, for the most part, work bestwith a clear frontal image of the face. However, in a real-life setting, a frontal viewof the face is not always available. Since a facial image acquired from a non-frontal
view reduces the accuracy of facial expression recognition, this type of system is not
optimal for real life tracking of facial expressions. Finally, a camera-based system does
not fit social contexts. Even though the problems above may be solved with a wearable
camera, such a device will likely stir public concerns for privacy. Also, since a camera
needs to be at a certain distance from the people to capture their facial expressions,this might interfere with a natural flow of personal interactions. Kimura et al. proposed an eyeglass-based, hands-free videophone to overcome these problems [Kimura
et al. 2013]. It can yield a frontal face image with fish-eye cameras and capture facial
expressions. However, the system is wired to laptop computers, which poses a major
problem in wearability.
ACM Transactions on Interactive Intelligent Systems, Vol. 9, No. 4, Article 39, Pub. date: March 2017.

39:4

K. Masai et al.

2.2. Facial Expression Recognition With Wearable Devices

One of the first attempts to detect facial expressions with a wearable device was Expression Glasses [Scheirer et al. 1999], which can recognize specific facial expressions(confusion/interest) by measuring facial muscle movement with piezoelectric sensors.
Gruebler and Suzuki designed a wearable device that can read positive facial expressions using facial EMG signals [Gruebler and Suzuki 2014]. Their device has to be
attached to the side of a face, but it can record the user’s affective state for more thanfour hours with high accuracy. It can be used during therapeutic interventions and tosupport medical professionals. Li et al. used a depth camera to capture expressions on
the lower half of the face, and eight strain gauges to capture expressions on the upper
half of the face inside a head-mounted display [Li et al. 2015]. They also mapped the
input signals to a 3D face model. These prior works used contact-base sensors. Whilethey performed well under experimental conditions, the measurement processes require continuous physical contact, meaning that the sensors/electrodes need to be attached to the user the whole time. This need for physical contact can make the user
experience rather uncomfortable, especially over a longer period of time.
2.3. Sensing with Photo Reflective Sensors

Fukumoto et al. used photo reflective sensors attached to the glasses to capture skindeformations at the corners of eyes and cheeks that occur with happy facial expressions [Fukumoto et al. 2013]. They then used threshold-based clustering to distinguishsmiles from laughs. While efficient, this method does not scale well for multiple users
because of the individual variations in determining the appropriate threshold (i.e. How
much the skin around the eyes moves while smiling or laughing varies from person toperson). Besides, due to the limited number of sensors, it can miscategorize other facial
expressions as the target ones. There are some other interesting applications involving
a limited number of photo-reflective sensors. For instance, Nakamura et al. proposed a
device with one photo reflective sensor, which intuitively and seamlessly controls augmented reality information using the natural movement of eyebrows when users try
to focus and stare at something [Nakamura and Miyashita 2010].
Photo-reflective sensors have also been used to capture skin deformations on various
parts of the human body aside from the face. For example, Ogata et al. leveraged skindeformations on the forearms to use skin as an interface (SenSkin) [Ogata et al. 2013].
This work used two arrays of six photo reflective sensors to detect gestures (pinch,
touch, pull-up, pull-down, pull-right, and pull-left) on the skin surface. Their previous
work (iRing) also leveraged skin deformations on the finger to capture finger gesturesand external input [Ogata et al. 2012].

3. APPROACH TO FACIAL EXPRESSION RECOGNITION

Eyewear computing is a promising technology for facial expression and affect recognition in real life. Since the head is the primary location for most of the human senses, we
can gain access to a variety of physiological signals by placing sensors in the head area.
There remains a design problem as anything worn on the head is quite noticeable, yetwith increasingly smaller Printed Circuit Boards (PCB), sensors, and actuators, we can
now build smart glasses that are similar in appearance from normal eyewear, making
them socially acceptable in terms of both appearance and comfort.
In this research, we use a large number of photo-reflective sensors and apply a machine learning method for measuring facial expressions. This approach has the following advantages: (1) An adaptive and robust facial expression recognition that works
with a variety of users, enabled by the richness of the sensor information and the
ACM Transactions on Interactive Intelligent Systems, Vol. 9, No. 4, Article 39, Pub. date: March 2017.

Evaluation of Facial Expression Recognition by A Smart Eyewear

39:5

Fig. 3. Skin deformations change the distances between the sensors and the skin surface. The deformations
occur when the facial expression changes.

distance

Fig. 4. (a)We measured the distance between a photoreflective sensor and skin surface. (b)PCB used forthis experiment.
Fig. 5. The voltage change of the photo reflective sensors related to skin surface distance.

machine learning framework. (2) Non-contact measurement: The sensors are unobtrusive and do not require physical contact, which improves the wearability of the device.
(3) Smart appearance: The sensors are small enough to be integrated into everyday
glasses (e.g., NJL5908AR by New Japan Radio Co., Ltd: 1.06 x 1.46 x 0.5 mm), making the device suitable for everyday usage. (4) Simplicity: The processing required tointerpret the sensor readings is minimal, so no elaborate feature extraction is necessary. We can, therefore, keep the processing cost and energy consumption little, whichis crucial for practical real-time recognition. (5) Affordability: The device only uses
photo-reflective sensors with a micro processing unit and can be manufactured at alow cost.
To capture facial expressions, we leverage skin deformations caused by the movement of facial muscles (Figure 3). When users move their facial muscles, threedimensional deformations occur on the skin surface. Each facial expression involves
different movements of facial muscles. The movements of the eyelids, the eyebrows,
the nose, and the cheeks all cause three-dimensional skin deformations around theeyes. The movement of the mouth also causes the skin deformation under the eyes
because the muscle movement around the mouth causes a cheek deformation that
extends to the area below the eyes. According to [Ekman and Friesen 1977], these
movements are the greater parts of Action Units (AUs) with which the Facial Action
Coding System codes human facial expressions. Therefore, placing sensors to capture
the skin deformations around the eyes makes it possible to detect most muscle movements related to the target facial expressions [Tian et al. 2001]. Skin deformations arecaptured by measuring the distances between the skin surface and the photo reflective
sensors embedded in the various spots of the eyewear device.
3.1. Photo Reflective Sensor

As discussed in Related Work, photo reflective sensors are sometimes used in thefield of Human Computer Interaction to measure human skin deformations [Fukumoto et al. 2013; Nakamura and Miyashita 2010; Ogata et al. 2013; Ogata et al. 2012].
ACM Transactions on Interactive Intelligent Systems, Vol. 9, No. 4, Article 39, Pub. date: March 2017.

39:6

K. Masai et al.

Fig. 6. Data processing pipeline.

Fig. 8. The placement of photo-reflective sensors

Fig. 7. System components

We use infrared (IR) reflective sensors for this project. The sensors are composed of an
IR LED and IR phototransistor. We used a 62K ohm resistor for the phototransistor of
the sensors and an 180-ohm resistor for the LED of the sensors.
To establish the basic characteristics of skin surface reflection captured by an IR
photo reflective sensor, we measured the voltage from the sensor (Figure 4). It changes
by the distance between the sensor and the skin surface. We collected 30 samples at
each position. Figure 5 shows the average and standard deviation at each distance.
The standard deviation is quite small (at most 0.014 V). The correspondence is not
linear. We can obtain the proximity to the skin. For closer distances, the photo reflective
sensors have a higher resolution.
4. IMPLEMENTATION

Figure 6 shows the data processing pipeline of our system. In this section, we describe
Hardware and Software implementation details.
4.1. Hardware

Figure 7 shows the components of our prototype. The prototype incorporates 17 photoreflective sensors (SG-105 by Kodenshi, the placement can be seen in Figure 8), a
ACM Transactions on Interactive Intelligent Systems, Vol. 9, No. 4, Article 39, Pub. date: March 2017.

Evaluation of Facial Expression Recognition by A Smart Eyewear

39:7

Fig. 9. User interface

16-channel multiplexer (CD74HC4067 by Sparkfun), a transistor (IRLU3410PBF by
International Rectifier), Arduino Fio, Xbee, and lithium polymer battery. The weightof the prototype is around 60g. The front frame is 3D printed, and the temple tips aretaken from a regular commercial eyewear. An eyewear band is added to stabilize theposition of the eyewear. The transistor is used to modulate the LED of the photo reflective sensors because the sensors are easily influenced by ambient light such as thefluorescent lighting in the environment. We measure the difference between the values
with LED on and off. The switching frequency is around 80 Hz. With this method, it ispossible to reduce the influence of ambient light. Xbee enables serial communication
via ZigBee at 57600 bits per second.
4.2. Software

In an Arduino environment, input from each sensor is converted into a 10-bit value. In
order to take the difference of sensor values between LED is on and off, we multiply
minus one to the value when LED is off. The value is smoothed out by applying a moving average to ten sequences of each sensor value to reduce noises. One data sample
is a collection of 17 elements from the sensors. Each element of the data sample is themoving average of ten sensor values. The data sample is then sent to Java/Processing.
In the Processing environment, we normalize the data sample and record the normalized data sample with a desired facial expression label as a training set. With thetraining set, we apply an SVM algorithm with a radial basis function (rbf) kernel (C= 10, gamma = 1.0) to classify facial expressions in real time. Figure 9 shows the visualization of our system. The emoticon shows the classification result. The bar graphrepresents the raw data. The yellow dots correspond to the positions of the sensors,
and the size of the dot is correlated to the normalized value of each sensor. We use only
one data sample for classification. For later experiments, we record normalized data
samples described in this section.
4.2.1. Machine Learning(1) When the user makes a neutral expression with the device on, we set the baselineof the elements in the data sample(BLV ) as 0.5.
(2) In this stage, the user dynamically moves his/her facial muscles. For each sensor,
the range of each sensor value (Range) in the data sample is determined during thiscalibration process. Based on these values, the normalized sensor value (N SV ) for
each sensor is calculated as follows. An element of the data sample, which is the
ACM Transactions on Interactive Intelligent Systems, Vol. 9, No. 4, Article 39, Pub. date: March 2017.

39:8

K. Masai et al.

moving average on each time frame, is defined as SensorInput. T olerance is chosen
as 40 experimentally.
Range = (M ax − M in) + T olerance(1)(
N SV = 0.5 + (SensorInput − BLV )/Range
if N SV > 1, then N SV = 1(2)
if N SV < 0, then N SV = 0
Range includes T olerance since there is a trade-off with relying only on the acquired data.
— Advantage: Normalization can set the appropriate range for each sensor by measuring the range of SensorInput that varies depending on the geometry of eachuser’s face and the position of the sensor. The normalization can improve robustness because it can accommodate the weight of each SensorInput.
— Disadvantage: The position of the device during calibration phase is not always
stable. The user may move his/her facial muscles too dynamically and cause
the eyewear to dislocate, resulting in inaccurate measurement of the Max and
Min values. On the other hand, if the facial movement during calibration is not
dynamic enough, it may reduce the amount of information that can be obtained.
Therefore, it is not always possible to normalize the data sample in an optimal
manner.
(3) During the learning phase, normalized data samples are stored with facial expression labels that are selected as the desired outputs. A label is attached to each data
sample.
(4) For real-time classification, the normalized data sample is applied to SVM that istrained with the normalized data samples with the labels.
4.2.2. Algorithm.

In addition to the training set that includes the output labels and the normalized data
samples, the calculated values (CV ) from two different sensors are also used for SVM.
The calculation formula for (Si ,Sj | 1 ≤ i, j ≤ 17 ) is shown below
CV = (Si − Sj )/2 + 0.5 if CV > 1, then CV = 1
if CV < 0, then CV = 0(3)

Sensor placement is shown in Figure 8. We considered the calculated values from adjacent sensors located in the bottom part of the front frame as well as the values fromthe sensors that have a vertical relationship. The calculated values from adjacent sensors in the bottom part are important because the surface is smooth and the skin isconsidered as an elastic model, thus they correlate with each other as discussed by
Ogata et al. [Ogata et al. 2013]. The data from the sensors that have a vertical relationship can partially inform the position of the eyewear and the face. In total, we
used 33 dimensions (normalized input: 17 + adjacent data: 7 [(10,11), (11,12), (12,13),(13,14), (14,15), (15,16), (16,17)] + vertical data: 9 [(1,10), (2,11), (3,12), (4,13), (5,14),(6,15), (7,16), (8,17), (9,17)]).
5. SYSTEM EVALUATION

For the system evaluation, we conducted four experiments. In the first experiment,
we recorded the test dataset immediately after user dependent training. The results
suggested the user dependency of our system. We then evaluated the trade-off between
the number of sensors and accuracy, the robustness to changes in head direction as well
ACM Transactions on Interactive Intelligent Systems, Vol. 9, No. 4, Article 39, Pub. date: March 2017.

Evaluation of Facial Expression Recognition by A Smart Eyewear

39:9

Fig. 10. Evaluation with different face directions

Fig. 11. Experimental Setting

as to the removal and remount of the device. In the second experiment, we had a trial
for multiple days to assess the possibility of long term usage. In the third experiment,
we tested the robustness of the system by taking measurements while walking. In
the final experiment, we evaluated the influence of the vertical displacement of ourdevice on recognition accuracy. We also describe observations from a demonstration at
SIGGRAPH Emerging Technologies 2015 [Masai et al. 2015].
5.1. Evaluation 1: Basic Setup

Eight users (four Japanese, one French, one Chinese, one Taiwanese, and one Sri
Lankan. Two of them female. Average age: 27.3) participated in our evaluation. They
were asked to sit in a chair and mimic the pictures of an American male (retrieved
from the images of a man from a TV show ”Lie to Me”) making the universal facial expressions based on Ekman et al. [Ekman 1989]. First, the user looked straight ahead
with a neutral facial expression, setting the baseline for the sensors. Then, the user
moved the facial muscles for the calibration. We collected the data samples of eightfacial expressions in different poses: Looking straight ahead (three times), looking up(three times), looking down (three times), looking left (two times), looking right (twotimes), and taking off the device and putting it back on (two times) (Figure 10). We collected data samples with different head directions because the movement of the headalone causes skin deformations as a result of the effects of gravity and joint coupling ofmuscles. The experimenter manually recorded ten data samples with facial expression
labels while the user kept their maximum pose of each facial expression at regular
50-millisecond intervals. Overall, each user?s dataset includes 1200 data samples( 10samples per expression per time * 8 facial expressions * 15 times in different poses.
All recordings were conducted indoors. (Figure 11).
5.1.1. Accuracy with User-Dependent Training.

Firstly, we tested the accuracy in measuring each user’s performance with user dependent training. For each dataset, one recording (10 data samples) of each facial expression was divided into two sets, with the former half as a training set ( 600 data samples:
5 data samples per expression per time * 8 facial expressions * 15 times in different
ACM Transactions on Interactive Intelligent Systems, Vol. 9, No. 4, Article 39, Pub. date: March 2017.

39:10K. Masai et al.

poses) and the latter half as a test set (600 data samples). We applied the training
set to SVM with a rbf kernel (C = 10, gamma = 1.0). In other words, we trained withuser A training set and tested with user A test set. We achieved 92.8% accuracy on
average (Facial-Expression-based result was 84.3% - 97.8%. User-based result : 84.8%- 99.2%.). By learning from the data obtained with different head directions, our device was able to classify the facial expressions correctly, regardless of where the head
was directed at the time of the measurement. Table I shows the confusion matrix ofthe results. As shown in the matrix, disgust can be similar to anger or fear. Besides,
surprise and fear are close facial expressions with a 4.7% error to each other.
Table I. Confusion matrix (within subjects)

Actual
Value

N
96.8%
0.3%
1.7%
0.3%
2.0%
1.5%
4.5%
1.5%

Neutral(N)Happiness(H)Disgust(D)Anger(A)Surprise(Su)Fear(F)
Sadness (Sa)contempt(C)

H
0.7%
98.3%
0.2%
0%
0%
0%
0%
0.3%

Classified Results
D
A
Su
0.2%
0%
1.2%
0.5%
0%
0%
84.3%
3.2%
0%
0.7% 96.5%
0.8%
1.2%
0.2% 91.2%
4.2%
0.8%
4.7%
2.5%
1.8%
0.8%
0%
0%
0%

F
1.0%
0%
5.5%
0.3%
4.7%
87.5%
4.7%
0.3%

Sa
0.2%
0%
3.7%
1.3%
0.8%
0%
85.7%
0%

C
0%
0.8%
1.5%
0%
0%
1.3%
0%
97.8%

5.1.2. User Dependency.

Secondly, we evaluated user dependency by training with each user’s training set andtesting with all other users’ test sets. For example, we trained with User A’s training
set and tested with User B, C,..., F’s test sets respectively. The results are shown in
Table II. When the training set and the test set are taken from different users, theaccuracy scores drops significantly: Accuracy is 48.0% at best (using the training datafrom User A and the test data from User G). This result indicates that, with the current
system, users need to calibrate the device individually for accurate classification of
their facial expressions.
Table II. User dependency matrix

Training
Data

User A
User B
User C
User D
User E
User F
User G
User H

A
99.2%
39.2%
42.5%
19.8%
23.3%
44.3%
43.8%
12.5%

B
36.8%
98.7%
22.3%
29.8%
27.3%
34.2%
31.0%
25.2%

C
31.8%
23.3%
84.8%
33.5%
18.7%
28.2%
24.7%
15.7%

Test Data
DE
34.5% 28.8%
13.5% 30.0%
37.7% 21.0%
85.0% 32.0%
20.8% 89.3%
27.8% 27.7%
20.0% 27.7%
14.8% 26.2%

F
40.5%
33.8%
37.3%
25.3%
17.3%
97.3%
28.8%
17.8%

G
48.0%
32.8%
40.7%
30.5%
39.8%
35.0%
88.7%
29.8%

H
28.3%
26.7%
22.2%
18.0%
39.5%
34.2%
28.5%
95.2%

Table III shows the confusion matrix of all the results. Though there is user dependency, happy expressions can be recognized with relatively high accuracy (71.9%). The
other facial expressions were harder to be recognized(e.g. sadness: 17.2%, contempt:
22.7%).
Figure 12 shows the distribution of data samples of each facial expression from User
A. The sensor numbers (1 through 17) corresponds with the numbers shown in Figure
8.
ACM Transactions on Interactive Intelligent Systems, Vol. 9, No. 4, Article 39, Pub. date: March 2017.

Evaluation of Facial Expression Recognition by A Smart Eyewear

39:11

Table III. Confusion matrix(between subjects)

ActualValue

Neutral(N)Happy(H)
Disgust(D)Angry(A)Surprise(Su)Fear(F)
Sad (Sa)
contempt(C)

N
74.9%
5.3%
15.8%
14.9%
27.1%
14.5%
27.8%
29.5%

H
1.1%
71.9%
7.5%
7.8%
3.2%
6.5%
8.6%
7.8%

Classified Results
D
A
Su
6.3%
1.3%
5.2%
2.6%
5.1%
0.4%
23.5% 12.2% 14.0%
13.4% 26.6% 13.8%
10.1%
4.1% 30.9%
11.4%
8.8% 17.3%
7.4%
6.4% 15.9%
8.6%
5.0% 11.7%

F
3.2%
7.7%
17.5%
15.5%
11.7%
27.7%
13.6%
11.2%

Sa
2.6%
2.8%
4.1%
4.9%
8.9%
5.5%
17.2%
3.7%

C
5.3%
4.2%
5.2%
3.1%
4.2%
8.2%
3.1%
22.7%

Fig. 12. Distribution of sensor values changes for each facial expression.

5.1.3. Number of Sensors.

Next, we evaluated the trade-off between accuracy and the number of sensors. We usedthe data sets from all eight users for this purpose. We merged eight users’ training sets
into one training set (4800 data samples) and one test set (4800 data samples) respectively. We applied SVM in the same way as we did in the Basic Setup Analysis. Fromeach data sample, we began by choosing one sensor value that had the best accuracy
based on the result of SVM. Next, we added another sensor value with the second-best
accuracy. We repeated the process until all values of 17 sensors were included. For thisanalysis, we only applied the normalized 17-dimensional data to SVM.
As shown in Figure 13, the experiment yielded 84.1% accuracy with 17 sensor values.
The accuracy improved with the addition of more sensor values. With 13 sensor values,
our system achieved more than 80% (81.0%) accuracy. Sensors 1, 9, 12, and 15 were leftout. Sensors 1 and 9, as well as 12 and 15, are symmetrically located at the opposite
ends of the frame. The further the sensors are from the center of the face, the greater
the distance between the eyewear frame and the skin surface on the face becomes. We
think the greater distance results in the sensors having less information because they
are more vulnerable to the noise of ambient light. Photo-reflective sensors work wellwhen the distance between the sensor and the target is less than 10.00 mm.
5.1.4. Accuracy when User Changes their Head Direction and Removes/Remounts the Device.

We have already shown that we can recognize facial expressions even when there arechanges in head direction and when the user removes and remounts the device byobtaining data at those conditions in the training phase. We also evaluated how thoseconditions influence accuracy using the same dataset as before. We trained with thedata samples obtained when the user looked straight ahead and tested with the data
samples obtained in other conditions. The result is shown in Table IV. Accuracy varies
among the users, but mostly it is between 50% - 60%, indicating the relative robustnessof the recognition system.
ACM Transactions on Interactive Intelligent Systems, Vol. 9, No. 4, Article 39, Pub. date: March 2017.

39:12K. Masai et al.

Fig. 13. Trade-off between recognition accuracy and number of sensors
Table IV. Accuracy by different conditions compared to looking straight ahead

User A
User B
User C
User D
User E
User F
User G
User H
Average

Upwards
86.7%
67.5%
35.4%
24.6%
44.2%
50.0%
40.4%
81.7%
53.8%

Down
72.9%
50.4%
66.7%
32.1%
47.9%
58.8%
45.8%
37.5%
51.5%

Condition
Left
Right
84.4% 93.8%
48.1% 56.9%
56.9% 46.2%
65.6% 28.1%
63.1% 58.1%
59.4% 66.9%
61.9% 43.8%
46.9% 29.4%
60.8% 52.9%

Take Off&ON
79.4%
45.6%
53.1%
64.4%
68.1%
40.0%
53.8%
55.0%
57.4%

Average
83.4%
53.7%
51.7%
43.0%
56.3%
55.0%
49.1%
50.1%

5.2. Evaluation 2: Reliability over TimeIn our second experiment, we collected data samples from three of the participants inthe first experiment on different days (The data samples were obtained in the lookingstraight position only). Users were asked to sit in a chair and put on the device. Afterthe calibration, we collected data samples of the eight facial expressions three times
each (240 data samples: 10 samples per expression per time* 8 expressions * 3 times)on each day. Like Evaluation 1, the experimenter recorded manually while the user
kept their maximum pose of each facial expression at regular 50-millisecond intervals.
We conducted the procedure on three different days, and so we acquired 720 data samples with facial expression labels from each user. We used the data obtained on two of
the three days as a training set (480 data samples) and the data from the remaining
day as a test set (240 data samples). The results are shown in Table V. The averaged
accuracy for the three users was 78.1%. By making bigger the size of the training set,
the repeatability can be ensured. It suggests the possibility of long-term usage. In the
confusion matrix, the most dominant error was classifying 33.3% of the anger casesand 23.3% of fear cases as disgust.
Table V. Accuracy on different days

User A
User E
User F

Day1
83.8%
72.9%
86.3%

Day2
91.7%
70.8%
69.6%

Day3
90.4%
69.2%
68.8%

Average
88.6%
71.0%
74.9%

ACM Transactions on Interactive Intelligent Systems, Vol. 9, No. 4, Article 39, Pub. date: March 2017.

Evaluation of Facial Expression Recognition by A Smart Eyewear

39:13

5.3. Evaluation 3: Usage during Walking

For this experiment, we evaluated the effect of walking on the recognition of facial
expressions because the activity may cause changes in the position of eyewear, which
affect sensor values slightly. We collected 480 data samples with facial expression labels from each user. Four users participated (two males and two females, two of themJapanese, one Chinese, and one German. Age range: 25-59). We collected 10 data samples for the eight facial expressions in the looking-straight position. We repeated theprocess three times (Dataset A: 240 data samples). This process follows that in Section
5.2. After the data samples had been collected in the stable position, users were asked
to walk along a corridor at a natural speed. We manually collected 10 data samples
for the eight facial expressions three times each while they were walking and holding their maximum pose for each facial expression at regular 50-millisecond intervals(Dataset B: 240 data samples). Carrying a laptop, the experimenter walked along witheach participant. The experimenter asked him/her to make all facial expressions one
by one. Soon after recognizing his/her maximum pose of each facial expression, theexperimenter recorded data samples with a laptop. We used Dataset A as a training
set and Dataset B as a test set. The result was an average accuracy of 73.2%, whichis slightly worse than the result found in Evaluation 2. We assume this is because
walking caused the device to shift its position.
5.4. Evaluation 4: Robustness to Positional Drift

To make our system robust, the noise by the positional drift of the eyewear should beconsidered as shown in the last evaluation. In our fourth experiment, we evaluatedthe robustness to the slipping of the device down the nose. We do not consider the slip
to the side because it should not be a major issue if the eyewear is properly fitted tothe user. On the other hand, the downward slip of the glasses is a common occurrence.
In this evaluation, sensor value distribution is different from other experiments as
we collect data samples on various levels of the positional drift (Levels). Hence we normalized the dataset based on the average and standard deviation of training datasets.
First, in order to examine the relationship between the distance and the sensor values, we measured the distance d between the yellow mark and the base of the wearer’s
ear shown in Figure 14 in seven different positions. The distance corresponds to thedegree of the positional drift. At the same time, we measured the sensor values ofneutral expression in each position. We applied principal component analysis (PCA) to
reduce the 17-dimensional sensor data to the one-dimensional data. The result can beseen in Figure 15 ( top: raw data, bottom: PCA value). There is a linear relationshipbetween the distance and the sensor values.

Fig. 14. The distance between the end of the yellow part and the base of the ear were measured

Next, in order to evaluate the effect of the slippage to the robustness of facial expression recognition, we collected 960 data samples with facial expression labels in total
ACM Transactions on Interactive Intelligent Systems, Vol. 9, No. 4, Article 39, Pub. date: March 2017.

39:14K. Masai et al.

Fig. 15. The distance and sensor values (neutral) change depending on the level of positional drift. top: raw
data, bottom: PCA(10 data samples * 8 facial expressions * 3 times * 4 positional drift levels) from five
participants (four male and one female). They are all Japanese graduate students aged
22 - 27. We followed the procedure as Evaluation 1 except that we collect data samples
at different levels instead of different poses. Level 1 is the base state where there isno slippage. The bigger the number of the Level is, the greater the degree of positional
drift. The Figure 16 shows the snapshots of User B with different Levels.
The Figure 17 and Figure 18 show the averaged sensor values of all facial expressions at different Levels of Users A and B. The sensor numbers correspond to the ones
shown in Figure 8. The Levels are marked by colors. As the degree of the slip is userdependent, the Levels are defined relatively. Sensor value distributions for Users C, D
and E are shown in Figure 19 - Figure 22. These figures focus on particular expressions at different Levels (neutral, happy, angry, surprise) from different users. Two
expressions (anger and surprise) from user E are included to show how the sensor
value distributions differ by expressions.
We evaluated the possibility of predicting facial expressions at one positional drift
level using the data samples taken at another Level. The training set includes data
samples from two trials at one certain Level (160 data samples: 10 data samples per
ACM Transactions on Interactive Intelligent Systems, Vol. 9, No. 4, Article 39, Pub. date: March 2017.

Evaluation of Facial Expression Recognition by A Smart Eyewear

39:15

Fig. 16. The positional drift level of the eyewear (Left: Level1, Right: Level4)

expression per trial * 8 expressions * 2 trials) while the test set includes data samples
from another trial (80 data samples: 10 data samples * 8 expressions * 1 trial) at one
Level. We applied the cross-validation method: we evaluated 3 test sets from 3 trials
at each Level. We trained with the training set of Level 1 and tested with the test setof Level 1-4 respectively. The process is repeated for the training sets from each Leveland each user. The SVM applied was different from the one previously used (linear
kernel, C = 500) because the SVM with rbf kernel did not perform well. The result
was then averaged for all users. The matrix can be seen in Table VI. At all Levels, theaccuracy of facial expression recognition was best when we used the training set andthe test set on the same Level (78.0% - 87.8%). The further the distance between theLevels we used for the training set and the test set become, the worse the accuracy. We
conclude it is hard to predict facial expressions when the slip of the glasses happens
without the dataset that includes the data samples at the level of the slip.

Table VI. Accuracy of facial expression recognition using
training set and test set on different Levels
Level 1
Level 2
Level 3
Level 4

Level 1
83.7%
46.5%
25.2%
23.0%

Level 2
39.6%
79.6%
42.9%
26.0%

level 3
23.7%
45.9%
87.8%
44.6%

level 4
21.8%
22.0%
38.7%
78.0%

We also evaluated the accuracy of facial expression recognition with the data samples of all Levels. From each participant’s dataset, the data samples taken in two of thethree trials (640 data samples: 10 data samples per expression * 8 facial expressions* 2 trials * 4 Levels) were merged as a training set and the data samples from theremaining trial was used (320 data samples) as a test set. We applied cross-validation
method with SVM (linear kernel, C = 500). The results are shown in Table VII. The average recognition rate was 86.8%. We also applied PCA to the training sets. Averaged
accuracy of facial expression recognition slightly improved to 87.7%. The best results
are shown with (User A:12, User B:13, User C:12, User D:15, User E:16) principal
components respectively. Even when the positional drift of the glasses happens, oursystem can recognize facial expressions with robustness by learning the data samples
taken at different positional drift levels.

Table VII. Accuracy of facial expression recognition using training set that includes all Levels
Result with 17 Sensors
Best result(PCA)

User a89.3%
91.0%

User b85.5%
86.7%

User c
84.2%
82.3%

User d82.1%
83.5%

User e92.8%
94.9%

Average
86.8%
87.7%

ACM Transactions on Interactive Intelligent Systems, Vol. 9, No. 4, Article 39, Pub. date: March 2017.

39:16K. Masai et al.

Fig. 17. Facial expression distribution on each Level Fig. 18. Facial expression distribution on each Level(User A)
(User B)

Fig. 19. Sensor value distribution on each Level(User C, Neutral)

Fig. 20. Sensor value distribution on each Level(User D, Happy)

Fig. 21. Sensor value distribution on each Level(User E, Angry)

Fig. 22. Sensor value distribution on each Level(User E, Surprise)

5.5. Demonstration at SIGGRAPH Emerging Technologies 2015

We have demonstrated our eyewear device at SIGGRAPH Emerging Technologies
2015. During the demonstration, we had more than 200 users from a various international background try on the device. As the demonstration proceeded, we came torealize that the size and shape of the eyewear had to be adjusted to each user for accurate recognition. Three major issues were present: (1) The device sometimes slipped
ACM Transactions on Interactive Intelligent Systems, Vol. 9, No. 4, Article 39, Pub. date: March 2017.

Evaluation of Facial Expression Recognition by A Smart Eyewear

39:17

out of place when some users changed facial expressions. The slippage changed theaccuracy of the recognition because the device relies on the proximity sensing between
the front frame of the eyewear and the skin surface on the face. (2) Some of the sensors positioned between users’ eyebrows saturated and did not work well for those who
had high nose bridges. This problem could occur even with the neutral expression. 3)
The sensors placed on the top can measure the distance to the eyelid or the eyebrows
depending on the shape of the users’ face or the position of the eyewear. Therefore,
the sensor data can be different for each user, requiring individual training and calibration. Due to these reasons, the eyewear should be customized to each user. We also
observed that the eyewear seemed to work better for Caucasians compared to Asians
as Caucasians tend to be more expressive.
6. INITIAL FIELD TRIALS

Fig. 23. Distribution of predicted result of facial expressions based on recorded sensor values for a longtime

We conducted initial field trials for daily life by recording facial expressions of auser in a daily living or home scenario. The recording was a 90 minutes time seriesconsisting of four activities: (1) playing Go game with a computer for 32 minutes (aboard game involving two players originated in China) (2)playing with a dog for 16
minutes (3) watching an episode of ”Friends” for 22 minutes and (4) programming on
a computer for 20 minutes. Additionally, playing a shooting game with friends (wonand lost: 10 minutes) and watching an episode of a crime drama ”Crime Scene Investigation (CSI)” for 45 minutes were recorded on the following day to compare (1)the activities include social interactions (human-human and human-animal) as well
as solitary activities, and (2) the same activities in a different context (e.g. winning
vs. losing games, watching comedy vs. drama series). Figure 23 shows the frequency
distribution of recognized facial expressions during the recording period. The figure
presents a normalized distribution for every 2 minutes with a logarithm contrast enhancement. The distribution of facial expressions varied depending on the activities.
For instance, happy expressions were mostly observed while interacting with the dog.
The sitcom also produced happy expressions between intervals of neutral ones. While
playing the Go Game, negative expressions sometimes occurred, but they were eventually replaced with a happy expression reflecting the progression of the game (fromfacing challenges to winning the game). During individual activities, the user tended
ACM Transactions on Interactive Intelligent Systems, Vol. 9, No. 4, Article 39, Pub. date: March 2017.

39:18K. Masai et al.

to show fewer facial expressions. For example, while programming induced some angry expressions, the dominant expression was neutral. On the other hand, the user
showed more various facial expressions (other than neutral) while interacting withthe dog. The user also showed various facial expressions while playing the shooting
game with a friend. The result is shown in Figure 24. During the game, the user displayed more various facial expressions such as happy and angry while playing withanother person, compared to when the user played alone with the computer (the Gogame). It suggests that social interactions induce more different facial expressions.
Although this is only a preliminary field trial, it is tempting to speculate that facial
expressions may be used as an indication of communication between people.
The distribution of facial expressions may also be influenced by the nature of theinteractions as seen in the case of gaming in the trial (Figure 25). When the user won,
he showed more happy expressions, especially in the latter half of playing. When theuser lost, negative expressions were displayed, the dominant expression being anger.
However, the user frequently displayed happy expressions during the gameplay evenwhen he eventually lost.
While watching the episode of the sitcom ”Friends,” the user sometimes showed
happy expressions. When the user watched an episode of the drama ”CSI” that included some graphic scenes such as murder, dissection of the human body and bleeding, disgust and surprise were detected. While this is an unsurprising result, it is once
again tempting to speculate whether analyzing the distribution of facial expressions
may be able to provide some feedback on the user experience.

Fig. 24. Facial Expression Ratio during Activities

7. DISCUSSION

In this paper, we presented a smart eyewear prototype that can recognize the wearer’s
facial expressions with 92.8% accuracy using user-dependent training in the experimental setting. This result shows the potential of our proposed approach. Thoughmany of the users remarked that they did not see clear differences between surpriseand fear, fear and disgust, and disgust and anger when they looked at the pictures
presented for instruction, It was possible to classify those expressions. The subtlety ofdifferences between some expressions such as anger and disgust may inevitably leadto miscategorization. However, we believe this owes more to the ambiguous natureof human facial expressions than to the design of our recognition system. We should
also note that there is some empirical evidence that challenges the universality of basic emotions (See [Russell 1994] for a discussion on cross-cultural recognition of facial
expressions).
Our analysis using the user dependency matrix showed the need for user-dependent
training. This does not seem to pose a major problem with our device since eyewear istraditionally a personalized item as pointed out by Scheirer et al. [Scheirer et al. 1999].
Our device can be designed as personalized eyewear, intended to be used by a single
ACM Transactions on Interactive Intelligent Systems, Vol. 9, No. 4, Article 39, Pub. date: March 2017.

Evaluation of Facial Expression Recognition by A Smart Eyewear

39:19

Fig. 25. Comparison between Activities

user. However, having external datasets may help reduce the cost of the learning phaseand improve the accuracy of recognition. Ideally, such external datasets should consistof data samples from a large number of users rather than a large volume of samples
from few users as suggested by Girard et al. [Girard et al. 2015].
The evaluation of our system found that accuracy was reduced in the case of longterm usage and walking. This reduction is due to the physical condition of the device
such as the effects of ambient light and positional drift. Another factor for the reduction in accuracy was the difficulty of reproducing the same facial expressions acrossdifferent times and conditions. It was quite challenging for the users to try to repeat
the exact same facial expression. They made slightly different facial expressions withvarying intensities. This is an inherent problem in developing a facial recognition system. However, our findings suggest that this may be made less of a problem by increasing the size of training sets with more trials. In our experiment that measured
the effects of positional drift, we were able to achieve an accuracy of 87.7% by learning
the data samples at different levels of positional drift. Requiring the users to conduct
extensive and repetitive training would probably be unrealistic for real-life usage, but
creating a new system that utilizes external training datasets may mitigate this issue.
We classified facial expressions by basic emotion categories. However, our facial expressions may not necessarily represent our inner emotions. Our face may reflect mental effort or convey a communication signal. Information other than basic emotions was
not considered in this paper, but some of the results may have been better understood
if we had ways to recognize such non-emotional signals. For example, in the field trials,”angry” facial expressions were recognized during computer programming or Go game.
It is not difficult to imagine that this was more a reflection of mental effort, confusion
or frustration rather than an expression of anger. Du et al. suggested the existence of
21 expressions [Du et al. 2014]. The information provided by complex facial expressions would be useful for understanding the user experience in depth. We regard ourwork as a first step towards recognizing more complex facial expressions.
8. LIMITATIONSIn this study, we assumed that skin deformations around the eyes indicate changes infacial expression. However, other behaviors such as yawning, rubbing one’s eyes, and
ACM Transactions on Interactive Intelligent Systems, Vol. 9, No. 4, Article 39, Pub. date: March 2017.

39:20K. Masai et al.

resting one’s cheek in one’s hand can also cause skin deformation. These normal behaviors can affect our system of facial expression recognition. Moreover, readings fromthe photo reflective sensors are affected by the condition of the facial skin. Factors suchas tanning, sweat, makeup, and facial swelling may require users to do calibration andtraining again.
The eyewear device only collects data from the sensors around the eyes. Although
movements of the mouth are partly detectable, there are several mouth movements
our system cannot detect because mouth movements and cheek deformations do nothave a one-to-one correspondence. For instance, the action of opening a mouth lowers
the cheeks slightly, which makes the distance between the eyewear frame and theskin surface greater. Therefore, when the sensor values on the lower part changed, oursystem cannot determine either movement of mouth or cheek deformation cause thechanges.
During the experiments, the participants made intentional facial expressions, although the goal of our work is to capture natural facial expressions. The posed andnatural expressions are similar to a certain extent, but there is no doubt some differences exist between them. Such differences can have an adverse influence on theaccuracy of natural facial expression recognition, especially because our natural facial
expressions are more subtle than posed ones. In the field trial, our device recognized
the expressions that were high in intensity, which suggests that our system were unable to pick up less intense expressions. That means our system can only provide anapproximate picture of the pattern of facial expressions in a daily life setting at thistime.
9. USAGE SCENARIOSIn this section, we would like to present a few speculative scenarios to explore someideas for future research. The first scenario is ”Collaborative Media Tagging” thatuses facial expressions or emotions to evaluate contents. The idea is to record andcompile facial expressions of the users on a large scale as they read books or watch
movies/videos, and index the contents so they can be searchable. The second is ”Care
System for Older Adults”. A wearable device that can provide an overview of the user’s
facial expression changes in their daily life could be used to increase awareness of theuser’s emotional state. We can imagine a system that would notify the children, nudging them to give their parents a call and talk to them when the system detects they are
feeling sad and smiling less. The third is ”Supporting System for People with Autism
Spectrum Disorders”. People with autism have difficulty in creating facial expressionsof emotion. Our system can help them create facial expressions by giving them a motivation to intentionally express their emotion with feedback if they could successfully
make the expressions or not. The fourth is ”Happiness Map”. If we get to the point
where we can reliably recognize facial expressions in our daily life, we could combine
facial expressions with location and demography. It may be possible to then search forplaces where the users frequently smile and laugh when comparing places to visit, live
or work.
10. CONCLUSION AND FUTURE WORK

We presented a novel smart eyewear that recognizes a wearer’s facial expressions.
With the eyewear device, we conducted an evaluation to recognize eight universal facial expressions. The experimental results showed recognition rates of 92.8% for onetime use regardless of facial direction or removal/remount of the device; 78.1 % forrepeatability and multiple-day usage after a training process; and 87.7% if we takethe positional drift of the glasses into account. The robustness in daily scenes can beachieved by learning more data. Moreover, we designed our prototype to be socially
ACM Transactions on Interactive Intelligent Systems, Vol. 9, No. 4, Article 39, Pub. date: March 2017.

Evaluation of Facial Expression Recognition by A Smart Eyewear

39:21

acceptable, following the looks of existing eyewear made possible by small photo reflective sensors. Our system still has room for improvement regarding calibration andaccuracy, yet we believe it is a major step in quantifying the flow of facial expressions
towards assessing human affection in daily life.
We regard the following four issues as our future work.
Firstly, we would like to design a natural learning process to capture natural facial
expressions. To this end, we plan to apply unsupervised learning method to the sensor
data collected in daily life settings. With this approach, we would not need any labelingfor sensor data, and categorizing natural facial expressions would be possible.
Secondly, we plan to design an optical filter to reduce the influence of ambient light.
The prototype used photo reflective sensors that function on the basis of IR reflection.
Because sunlight contains enormous amounts of IR light, sensor data saturates when
the sensors are exposed to direct sunlight. With the current system, the sensors are
not covered by anything and are easily influenced by an intense ambient light eventhough we applied light modulation to the LED of the photo reflective sensors.
Thirdly, we are looking to improve the calibration process. For robust recognition,
the prototype requires each user to calibrate under various situations. We want to
reduce this process by generating an additional dataset based on already trained data.
We will also consider transferring the learning method so that we can use other users’
data sets to calibrate another?s effectively.
Finally, we would, of course, like to try to detect more subtle facial expressions. We
used a classification algorithm for the prototype, and the intensity of facial expressions was not measured. This meant the current system could not measure complicated facial expressions such as ones that contain both happiness and surprise. Since
our real-life facial expressions are not limited to the 8 categories we considered in
the present study, we need to work on recognizing more complicated or subtle facial
expressions. Also, we think physiological information such as skin temperature, skinconductance, and eye gaze may be able to provide additional information that can contextualize some of the more ambiguous facial expressions. These sensory inputs could
be integrated into the current form of the system. We also believe that photo reflective sensors can detect skin deformations caused by movements of eyes. By applying atime-series data processing algorithm, we should be able to detect the four directionsof eye movements and blinks. Ultimately, our distant goal is to detect facial expressions related to cognitive loads such as attention, interest, fatigue and concentrationby combining these different types of information.

ACKNOWLEDGMENTS
The authors would like to acknowledge the support from JST CREST.

REFERENCES
A. Calvo, R. and S. D’Mello. 2010. Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications. Affective Computing, IEEE Transactions on 1, 1 (Jan 2010), 18–37.
DOI:http://dx.doi.org/10.1109/T-AFFC.2010.1
S. Du, Y. Tao, and M. Martinez, A. 2014. Compound facial expressions of emotion. Proceedings of the National
Academy of Sciences 111, 15 (2014), E1454–E1462. DOI:http://dx.doi.org/10.1073/pnas.1322355111
P. Ekman. 1989. The argument and evidence about universals in facial expres-sions. Handbook of social
psychophysiology (1989), 143–164.
P. Ekman and V. Friesen, W. 1977. Facial action coding system. (1977).
K. Fukumoto, T. Terada, and M. Tsukamoto. 2013. A Smile/Laughter Recognition Mechanism for Smilebased Life Logging. In AH. ACM, New York, NY, USA, 8. DOI:http://dx.doi.org/10.1145/2459236.2459273

ACM Transactions on Interactive Intelligent Systems, Vol. 9, No. 4, Article 39, Pub. date: March 2017.

39:22K. Masai et al.

M. Girard, J., F. Cohn, J., A. Jeni, L., S. Lucey, and D. Torre, F. 2015. How much training data for facial
action unit detection?. In 2015 11th IEEE International Conference and Workshops on Automatic Faceand Gesture Recognition (FG), Vol. 1. 1–8. DOI:http://dx.doi.org/10.1109/FG.2015.7163106
A. Gruebler and K. Suzuki. 2014. Design of a Wearable Device for Reading Positive Expressions
from Facial EMG Signals. Affective Computing, IEEE Transactions on 5, 3 (July 2014), 227–237.
DOI:http://dx.doi.org/10.1109/TAFFC.2014.2313557
T. Kanade, F. Cohn, J., and YingLi T. 2000. Comprehensive database for facial expression analysis. In Automatic Face and Gesture Recognition, 2000. Proceedings. Fourth IEEE International Conference on.
46–53. DOI:http://dx.doi.org/10.1109/AFGR.2000.840611
D. Keltner, P. Ekman, G. C. Gonzaga, and J. Beer. 2003. Expression of emotion. Handbook of Affective
Sciences (2003), 411–414.
S. Kimura, M. Fukuomoto, and T. Horikoshi. 2013. Eyeglass-based Hands-free Videophone. In Proceedingsof the 2013 International Symposium on Wearable Computers (ISWC ’13). ACM, New York, NY, USA,
117–124. DOI:http://dx.doi.org/10.1145/2493988.2494330
L. Knapp, M., A. Hall, J., and G. Horgan, T. 2013. Nonverbal communication in human interaction. Cengage
Learning.
H. Li, L. Trutoiu, K. Olszewski, L. Wei, T. Trutna, P. Hsieh, A. Nicholls, and C. Ma. 2015. Facial Performance Sensing Head-mounted Display. ACM Trans. Graph. 34, 4, Article 47 (July 2015), 9 pages.
DOI:http://dx.doi.org/10.1145/2766939
G. Littlewort, I. Fasel, M. S. Bartlett, and J. R. Movellan. 2002. Fully automatic coding of basic expressions
from video. University of California, San Diego, San Diego, CA 92093 (2002).
K. Masai, Y. Sugiura, M. Ogata, K. Kunze, M. Inami, and M. Sugimoto. 2016. Facial Expression Recognition
in Daily Life by Embedded Photo Reflective Sensors on Smart Eyewear. In Proceedings of the 21stInternational Conference on Intelligent User Interfaces (IUI ’16). ACM, New York, NY, USA, 317–326.
DOI:http://dx.doi.org/10.1145/2856767.2856770
K. Masai, Y. Sugiura, M. Ogata, K. Suzuki, F. Nakamura, S. Shimamura, K. Kunze, M. Inami, and
M. Sugimoto. 2015. AffectiveWear: Toward Recognizing Facial Expression. In ACM SIGGRAPH
2015 Emerging Technologies (SIGGRAPH ’15). ACM, New York, NY, USA, Article 4, 1 pages.
DOI:http://dx.doi.org/10.1145/2782782.2792495
David Matsumoto. 1992. More evidence for the universality of a contempt expression. Motivation and Emotion 16, 4 (1992), 363–368. DOI:http://dx.doi.org/10.1007/BF00992972
A. Mehrabian. 1972. Nonverbal communication. Transaction Publishers.
H. Nakamura and H. Miyashita. 2010. Control of Augmented Reality Information Volume by Glabellar
Fader. In Proceedings of the 1st Augmented Human International Conference (AH ’10). ACM, New York,NY, USA, Article 20, 3 pages. DOI:http://dx.doi.org/10.1145/1785455.1785475
M. Ogata, Y. Sugiura, Y. Makino, M. Inami, and M. Imai. 2013. SenSkin: Adapting Skin As a Soft Interface.
In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST’13). ACM, New York, NY, USA, 539–544. DOI:http://dx.doi.org/10.1145/2501988.2502039
M. Ogata, Y. Sugiura, H. Osawa, and M. Imai. 2012. iRing: Intelligent Ring Using Infrared Reflection. In
Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST ’12).
ACM, New York, NY, USA, 131–136. DOI:http://dx.doi.org/10.1145/2380116.2380135
R.W. Picard and J. Healey. 1997. Affective wearables. Personal Technologies 1, 4 (1997), 231–240.
DOI:http://dx.doi.org/10.1007/BF01682026
Rosalind W Picard. 1995. Affective computing. (1995).
A. Russell, J. 1994. Is there universal recognition of emotion from facial expression? A review of the crosscultural studies. Psychological Bulletin 115 (1994), 102–141.
J. Scheirer, R. Fernandez, and R. W. Picard. 1999. Expression Glasses: A Wearable Device for Facial Expression Recognition. In CHI ’99 Extended Abstracts on Human Factors in Computing Systems (CHI EA’99). ACM, New York, NY, USA, 262–263. DOI:http://dx.doi.org/10.1145/632716.632878
Y. Tian, T. Kanade, and F. Cohn, J. 2001. Recognizing Action Units for Facial Expression Analysis. IEEE
Trans. Pattern Anal. Mach. Intell. 23, 2 (Feb. 2001), 97–115. DOI:http://dx.doi.org/10.1109/34.908962
Y. Tian, T. Kanade, and F. Cohn, J. 2011. Facial Expression Recognition. In Handbook of Face Recognition, S. Z. Li and A. K. Jain (Eds.). Springer London, 487–519.
DOI:http://dx.doi.org/10.1007/978-0-85729-932-1 19
Z. Zeng, Roisman G. I. Pantic, M., and S. Huang, T. 2009. A Survey of Affect Recognition Methods: Audio,
Visual, and Spontaneous Expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence31, 1 (Jan 2009), 39–58. DOI:http://dx.doi.org/10.1109/TPAMI.2008.52

ACM Transactions on Interactive Intelligent Systems, Vol. 9, No. 4, Article 39, Pub. date: March 2017.