AffectiveWear: Toward Recognizing Facial Expression
Katsutoshi Masai1*, Yuta Sugiura2, Masa Ogata1, Katsuhiro Suzuki1, Fumihiko Nakamura1,
Sho Shimamura1, Kai Kunze1, Masahiko Inami1, Maki Sugimoto1
1
Keio University ,2National Institute of Advanced Industrial Science and Technology (AIST)

(a)
(c)
(b)
Figure 1: AffectiveWear system (a), Principle (b), Eyewear (c), HMD (d)(d)

Introduction
Facial expression is a powerful way for us to exchange
information nonverbally. They can give us insights into howpeople feel and think. There are a number of works related tofacial expression detection in computer vision. However, mostworks focus on camera-based systems installed in theenvironment. With this method, it is difficult to track user’s face ifuser moves constantly. Moreover, user’s facial expression can berecognized at only a limited place.
We present the eyewear that can detect facial expression anytime,anywhere (Figure 1 a). This eyewear can categorize 7 facial
expressions by measuring the distance between an eyewear frame
and a skin surface of a person’s face with 8 photo reflectivesensors. Recognizable states are as follows: neutral, smile, laugh,disgust, angry, sad and surprise. With our method, an individual
difference can be ignored with user-dependent training. Several
works show the wearable systems that can recognize facialexpression. Yet, these works focus on detecting only one specificfacial expression. Our contribution is detecting 7 facial expression
states in daily life. With our device, user can better understandtheir mind, and computing systems can tap into the rich set of
information provided by nonverbal communication.
Implementation
HARDWARE. We have developed the AffectiveWear (Figure 1c) to detect 7 facial expressions. (Figure 2) This system includes 8
photo reflective sensors (SG-105), 160 - 270 ohm resistors for
LED and 62k ohm resistors for transistor, Arduino Fio, Li-Po
Battery and ZigBee. The sensors measure the distance between theeyewear and skin surface on face from 8 points. This distancechanges according to the movement of the facial muscles (Figure 1b). The placement of the sensors is chosen to pick up facialchanges in the action units related to the facial expressions wewant to recognize. We also implemented HMD type with 7sensors (Figure 1 d).
SOFTWARE. We implemented a Support Vector Machine(SVM) model for the categorization in Java/Processing.
1) The standard of sensor data is set when user wears our device
and makes neutral facial expression.
*e-mail: masai@kmd.keio.ac.jp
Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor commercial advantage and that copies bear this notice and the full citation on thefirst page. Copyrights for third-party components of this work must be honored. For allother uses, contact the Owner/Author.
Copyright is held by the owner/author(s).
SIGGRAPH 2015 Posters, August 09 – 13, 2015, Los Angeles, CA.
ACM 978-1-4503-3632-1/15/08.
http://dx.doi.org/10.1145/2787626.2792632

Figure 2: Variation of recognizable facial expressions2) User normalizes the range of the movement of facialmuscles by dynamically moving facial muscles.
3) The dataset for each facial expression is created while user
makes the same facial expression as the one our user interfacedisplays. These dataset is used as training data of SVM model.
4) User checks if facial expression is categorized correctly. If not,more training is employed.
User Experience
Our system lets the user gain insights about their facial
expressions over days (How often did I smile this week versus
last?). Tracking these changes enables users to understand more
about their frequent unintentional non-verbal clues, helping themto improve their communication skills. Users suffering fromdepression or other mental disorders might get indications if theirstate is improving using our system.
With our device, user can put emoticon or change their typographyof text messages according to facial expression while they aretexting. Since typography has the power to change the impressionof messages, this can enrich the communication with texts.
Straightforward application also includes tagging the facial
expressions to images or movies captured using a life-logging
system, e.g. the Narrative Clip or a GoPro.
Regarding business aspect, we expect that a cinema 3d glasses
with our technology could record the facial expression of theaudience estimating if specific scenes evoke the wanted effects.
With HMD equipped with our technology, user can reflect theirfacial expression on their avatar in virtual world. With Kinect, user
also can reflect their movement. People can enjoy more natural,
subtle ways of communication in virtual world.
Acknowledgements
This research was supported by CREST, JST.
References
FUKUMOTO, K., TERADA, T., TSUKAMOTO, M. A Smile/Laughter Recognition Mechanism for Smile-Based Life
Logging.
In Proceeding of AH 2013, ACM, pp. 213–220.