Interactivity

CHI 2015, Crossings, Seoul, Korea

Smart Eyewear for Interaction andActivity Recognition
Shoya Ishimaru

Yuji Uema

Abstract

Graduate School of Engineering
Osaka Prefecture University
Sakai, Osaka, Japanishimaru@m.cs.osakafu-u.ac.jp

Graduate School of Media Design
Keio University
Yokohama, 223-8526 Japan
uema@kmd.keio.ac.jp

Kai Kunze

Koichi Kise

Smart glasses and, in general, eyewear are a fairly novel
device class with a lot of possibilities for user interaction design and unobtrusive activity tracking. In thispaper we show applications using an early prototype ofJ!NS MEME, smart glasses with integrated electrodes todetect eye movements (Electrooculography, EOG) andmotion sensors (accelerometer and gyroscope) to monitor head motions. We present several demonstrations:
We show a simple eye movement visualization, detecting left/right eye motion and blink. Additionally, users
can play a game, ’Blinky Bird’. They need to help a bird
avoid obstacles using eye movements. We implemented
online detection of reading and talking behavior using a
combination of blink, eye movement and head motion.
We can give people a long term view of their reading,talking, and also walking activity over the day.

Graduate School of Media Design Graduate School of Engineering
Osaka Prefecture University
Keio University
Sakai, Osaka, Japan
Yokohama, 223-8526 Japan
kise@cs.osakafu-u.ac.jp
kai@kmd.keio.ac.jp

Katsuma Tanaka

Masahiko Inami

Graduate School of Engineering
Osaka Prefecture University
Sakai, Osaka, Japankatsuma@m.cs.osakafu-u.ac.jp

Graduate School of Media Design
Keio University
Yokohama, 223-8526 Japaninami@kmd.keio.ac.jp

Author Keywords
Permission to make digital or hard copies of part or all of this work forpersonal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. Copyrights for third-party
components of this work must be honored. For all other uses, contact theowner/author(s). Copyright is held by the author/owner(s).
CHI’15 Extended Abstracts, April 18–23, 2015, Seoul, Republic of Korea.
ACM 978-1-4503-3146-3/15/04.
http://dx.doi.org/10.1145/2702613.2725449

Eye Movement Analysis; Electrooculography; Eye wear;
Smart Glasses; Eye Wear Computing; Activity Recognition

ACM Classification Keywords
I.5.4 [PATTERN RECOGNITION Applications]: Signal
processing;307

Interactivity

CHI 2015, Crossings, Seoul, Korea

Motivation
Research and industry get more and more interested insmart eyewear, from Google Glass, over Epson Moverio,
the Oculus Rift to the Sony Smart Glasses. Yet, most ofthese designs still look a bit clunky and emphasize displays (augmented or virtual reality), not the sensing aspect. We believe in the potential for unobtrusive smart
glasses that focus on sensing physiological signals. Especially eye movement seems a compelling sensing modality. Although there are a lot of interesting cues and activities we can recognize using gaze, there are only few
researchers focusing on tracking eyes in realistic environments with minimal augmentation [1]. So far, eye movement analysis either required expensive mobile or stationary eye-trackers, googles or ear pieces with electrodes
touching the skin around the eyes [7, 1, 5]. Although wesee more and more cheaper optical eye tracking solutions,they still look obtrusive compared with standard glasses
and have battery limitations (cannot be used for morethan 1-3 hours).
Figure 1: Users wearing theMEME glasses.

We present several demonstrations using a first prototype of unobtrusive, affordable EOG glasses called J!NSMEME. We show the feasibility to detect simple eye
movements, apply them to a simple game and use theglasses to recognize reading/ talking, as well as long term
behavior detection to give people an quantified overview
about their cognitive, social and physical life.

stream the data to a PC, laptop, smartphone or tablet.
The electrodes are sampled with over 100 Hz, the motion sensor with over 50 Hz. Using these settings we already achieve a battery runtime of over 8 hours for thecurrent version (will be improved). However, it is to notethat we are using an early prototype of MEME, not the
final product. Before recording with the device the firsttime, the electrodes should be adjusted a bit to the user’s
nose/eyes to get an optimal EOG signal (especially forthe vertical signal component). This is a one-time adjustment due to the early prototype stage.

Eye Movements for User Interactions
Using the current prototype we can detect and display
eye movements. We focus on left and right movements
and eye blinks, as the up and down directions are a bituser-dependent (initial user adjustment of the glassesrequired). Left, right movements and blinks can be recognized and chained as input gestures for actions (e.g.
left-right-blink for accepting a notification/call versus
right-left-blink for dismissing it).Users can try to maketheir own action sequence.
Based on Blink detection, we created a simple game seenin Figure 3. ”Blinky Bird” based on an open-source version of ”Flappy Bird” allows the user to control a small
bird using eye blinks. Every time he closes his eyes, thebird flaps its wings. The user needs to avoid obstacles byblinking at the right time.

MEME Prototype Hardware
Figure 2: The glasses prototype:
There are the 3 electrodes
around the nose used for eye
movement analysis (EOG).

We implement our demonstrations on a J!NS MEME prototype shown in Figures 1 and 2. The prototype is unobtrusive and looks close to normal glasses. It is equippedwith 5 electrodes (3 around the nose, 2 on the ears) todetect eye movements and motion sensors (accelerometer and gyroscope), as well as a Bluetooth LE module toReading/Talking Detection
We also show reading and talking recognition using thedevice. Based on previous work, it is possible to classify
different activities offline using just blink detection andhead motion [2]. In previous work we used Google Glassfor the data recordings. The infrared distance sensor on308

Interactivity

CHI 2015, Crossings, Seoul, Korea

Google Glass can be used for blink detection and it comesalso equipped with motion sensors. We extend this work,and implement an online reading/ talking detection using
MEME.

Figure 3: User playing ”BlinkyBird” using eye blinks tomaneuver the small bird sprite.

Method – Users can wear the glasses and can get instant feedback (1-2 seconds delay) from a smartphoneor laptop computer what activity they are currently involved in. We recognize talking, reading and ”None” (forthe Null class) using head motion (variance of the 3 axis
accelerometer norm), blink frequency, as well as varianceof the horizontal eye movement component as features
for a j48 decision tree algorithm. The instant feedback
might not be so interesting for applications (except for realtime activity sharing or similar), yet gives us the ability
to record reading/communication habits for the day andmakes it easy to evaluate the performance of MEME.
Evaluation – We tested a prototype of our reading/talking detection system on 12 people. They performed
15 instances of each reading and talking as well as a 5-6
instances of other unrelated activities ( drinking water,eating etc.). Only 16 instances were wrongly classifiedleaving us with 91 % of accuracy. No instances of otheractivities were classified wrongly, indicating that our online system works as well as the offline classifier implemented on Google Glass data [2].

Figure 4: Histograms for thenumber of blinks during reading
and talking.

Long Term Behavior Recording
As mentioned before, reading and talking detection canprovide users with a quantified review of their day. We
think this application is mostly interesting for education
scenarios. Users will be encouraged to read more when
they get feedback on their reading volume. The more weread the higher are our vocabulary and critical thinking
skills [3, 9]. Getting an overview about talking activities

might be also useful for users leaning towards depression
just to name one example [8]. Increasing direct communication can also fend off anxiety in some cases[6].
Figure 6 depicts a user reading a book, while MEME
tracks his activity and shows how many words the user
read. The current word count implementation is based ontime alone, yet we can improve the accuracy taking someeye movement features into account. Figure 7 shows twopeople engaged in a conversation. One of them wears a
MEME tracking the time he talks.

Figure 5: Overview of the daily activities: reading volume
depicted as blue bars, talking as green lines and step count inred.

We plan on giving people an quantified overview about
their cognitive, social and physical life. Currently we implemented long term reading detection (word count basedon time), talking detection and a simple step counter using the accelerometer. People can get a summary of their
daily activity using a web front-end seen in Figure 5). Infuture, we want to extend this work along the cognitiveand social axes.

309

Interactivity

Figure 6: User reading, while
MEME is tracking the number ofwords read using time.

Figure 7: User taking, while
MEME is tracking the timetalked.

CHI 2015, Crossings, Seoul, Korea

Conclusion and Future Work

References

We presented interaction and recognition demonstrations
using an unobtrusive EOG glasses prototype. The applications are meant to show the potential of the device
category. In the following we will give a small outlook on
how we plan to extend our work.

[1] Bulling, A., Ward, J. A., Gellersen, H., and Tröster,G. Eye Movement Analysis for Activity Recognition
Using Electrooculography. IEEE Trans. PAMI 33, 4(Apr. 2011), 741–753.
[2] Ishimaru, S., Kunze, K., Kise, K., Weppner, J., Dengel, A., Lukowicz, P., and Bulling, A. In the blinkof an eye: combining head motion and eye blink frequency for activity recognition with google glass. In
Augmented Human, ACM (2014), 15.
[3] Just, M. A., and Carpenter, P. A. The psychology ofreading and language comprehension. Allyn & Bacon,1987.
[4] Kunze, K., Iwamura, M., Kise, K., Uchida, S., andOmachi, S. Activity recognition for the mind: Towarda cognitive quantified self. Computer 46, 10 (2013),0105–108.
[5] Kunze, K., Katsutoshi, M., Uema, Y., and Inami, M.
How much do you read? – counting the number ofwords a user reads using electrooculography. In Accepted at Augmented Human’15, ACM (2015), 1073–1078.
[6] Landman-Peeters, K., Hartman, C. A., van derPompe, G., den Boer, J. A., Minderaa, R. B., andOrmel, J. Gender differences in the relation betweensocial support, problems in depression and anxiety.
Social Science & Medicine 60, 11 (2005), 2549–2559.
[7] Manabe, H., and Fukumoto, M. Full-time wearable
headphone-type gaze detector. In CHI’06 Extended
Abstracts, ACM (2006), 1073–1078.
[8] Ross, C. E., and Mirowsky, J. Explaining the social
patterns of depression: Control and problem solving–or support and talking? Journal of health and social
behavior (1989), 206–219.
[9] Stanovich, K., and Cunningham, A. What reading
does for the mind. Journal of Direct Instructions 1, 2(2001), 137–149.

Word Counting– Our word counting algorithm for realtime feedback is so far very simple. We just take into account the time a user reads and estimate the word countaccordingly to the duration, not regarding if he reads fastor slow. To improve this we can use features derived fromthe horizontal component of the EOG signal.
Eye Gestures– We show the feasibility of detecting eye
gestures with our prototype. Yet, we have not designed
application cases and suitable eye gestures people mightwant to use, this is left for future work.
Cognitive Activity Recognition – We want to present
users with a comprehensive overview of their cognitiveand social activities. With quantifying them, we hope tolearn more about healthy habits to improve our mental
life and to induce positive long term behavior change [4].

Acknowledgements
Thanks a lot to J!NS for supplying us with prototypes.
This is work is partly supported by the CREST Project”Development of Fundamental Technologies for Innovative Use of Character/Document Media and Their Application to Creating Human Harmonized Information Environment”.

310