2017 International Symposium on Ubiquitous Virtual Reality

Open Smart Eyewear: From Sensing to Human
Amplification and Scaffolding Technologies
Kai Kunze
Graduate School of Media Design
Keio University, Tokyo, Japan
Email: kai@kmd.keio.ac.jp

Abstract—Human Attention is a finite resource and we needbetter tools to measure and manage it. We suggest an OpenEyewear Platform, a toolset to quantify social and cognitive
functions of individuals and groups of people in real life settings
using physiological sensing (e.g. eye movements, blink patterns,facial expressions etc.). This paper motivates smart sensing
eyewear as an enabling technology for various application cases
from behavior quantification to Human sensory amplification andscaffolding technologies.

I. I NTRODUCTION
In an increasingly interconnected world, and with a greaterflow of information, more people are having problems withkeeping attention and focus. Attention is a finite resource, andwe need to use it smartly. We need new tools to better manage
our attention, to improve our collective intelligence. There arepatterns in human physiological signals (facial expressions,heart rate, nose temperature, eye movements, blinks etc.) thatcan reveal information about intentions and cognitive functionsof an individuals and groups. So far, this data is only heavily
exploited by advertisement and marketing companies. This
proposal aims at exploring these patterns using an Open Eyewear Platform to better understand our behavior. So, people
can optimize for their own personal and society goals. We
see this as an integral first step to democratize Big Personal
Data and giving the user more control over their lives, livingwith greater awareness and intention, and giving people more
power to achieving their potential. We need better tools for
personal and society introspection.
I suggest an Open Eyewear Platform, a toolset to quantify
social and cognitive functions of individuals and groups ofpeople in real life settings using physiological sensing (e.g.
eye movements, blink patterns, facial expressions etc.) [1], [2].
For example, quantifying the attention level of students while
listening to a lecture, can give a lecturer information about
effective teaching methods and the individual student a way tointrospect and assess his efforts (also compared to his fellowstudents). After we can quantify these social and cognitive
functions, we want to design interactions/ interventions toimprove behavior.
In a second step, I also believe that we can use Smart
Eywear to amplify our senses. Given theFig. 1. Smart eyewear to measure attention and concentration: J!NS MEME
commercial sensing glasses.

interaction especially for tracking and enhancing social andcognitive functions [3], [4]. In contrast to Google Glass andearly wearable system approaches that just attached devices
to standard glasses, we envision exploiting smart eyeglasses
for assistance applications through fully integrated sensing,processing, and interaction functions in the regular spectacles
design, thus maximizing unobtrusiveness [5].
III. A PPLICATION A REAS
Smart Sensing Eyewear enables a couple of interesting
application areas. From ”simple”, ubiquitous behavior quantification to human sensory amplification and scaffolding technologies.
a) Behavior Quantification: focuses on moving socialand cognitive analysis from the lab to real life, as well as
giving non-technical users a way to visualize the data tobetter understand their behavior (see Figures 2 1). This
phase explores behavior patterns and physiological signals
indicative of social and cognitive functions on an individualand group level [6]. There is a lot of related work in cognitive
science and psychology, however most research focuses oncontrolled lab experiments with relative obtrusive equipment
and simple tasks (e.g. remembering random wordlists, simple
calculations).
b) Human Sensory Amplification: focuses on intuitive
interfaces for digital sensing and can also benefit from smart
eyewear as a baseline technology. Current sensing technologies
complement and exceed the capabilities of the human. ForII. A PPROACH
As humans receive most of their sensory input via the head,
it is a particularly interesting body location for sensing and978-1-5386-3091-4/17 $31.00 © 2017 IEEE
DOI 10.1109/ISUVR.2017.2242

Fig. 2. From obtrusive lab experiments to unobtrusive multipurpose sensing glasses.

example, the visible spectrum of a hum is only a fractionof what a CMOS chip can recognize. Although we havethese advances, there is very little research on how to use
those sensors in everyday tools carried on our body. We havethese set of analog wearable tools (e.g. binoculars) and wealready carry digital tools with us (e.g. smart phone). However,their use is by far not intuitive. In this case, intuitive meansthat we could use additional sensory modalities as extensionsof our body. Smart eyewear can help explore physiologicalsignals and other clues about the intentions of a user to makethese interactions as seamless as possible, creating superhumansenses.
c) Scaffolding Technologies: train a user to performbetter at a specific task with technology. However, the skills
the user train also increase when removing the technology. The
digital assistants are used only for a training phase and boost
the physical or cognitive baseline performance of the user [7],[8]. Especially for cognitive skill sets, eyewear might be agreat scaffolding tech [9]. There are indications that we canrecognize cognitive load and alertness using sensors embedded in unobtrusive eyewear, enabling novel interventions andinteractions for scaffolding during learning and teaching.
Major challenges for these research directions can be summarized as follows. For behavior quantification, establishingclear links between cognitive states and social behavior in
everyday life is difficult, especially targeting higher levelcognitive functions. The most challenging aspects of Sense
Amplificiation lei in interfacing digital technology with oursenses. Whereas one of the most critical issues for scaffolding
is how transferable knowledge and skill sets are between people and how much the learning experience is individualised.
IV. C ONCLUSION
Overall, I believe that smart eyewear can be an impressive
enabling technology from quantifying behavior in real life
to creating superhuman senses, systems that extend humansensory capabilities in an intuitive way using digital technology. Such extension would be continuously available andtransparent in use. A lot of digital sensor technology issuperior to human senses and we employ them already intools (e.g. heat cameras, radiation sensors, x-ray). However,their use is by far not straightforward. Affordances of usingthese tools in everyday life are still too high. We can use
physiological signals and sensing integrated in the glassses tounderstand the intent of users and lower these affordances.
ACKNOWLEDGMENT
This research is supported by JST PRESTO Open Eyewear
Project Grant Number JPMJPR16D4.
R EFERENCES[1] S. Ishimaru, K. Kunze, Y. Uema, K. Kise, M. Inami, and K. Tanaka,“Smarter eyewear: Using commercial eog glasses for activity recognition,” in UbiComp, ser. ’14 Adjunct. New York, NY, USA: ACM, 2014,
pp. 239–242.
[2] S. Ishimaru, K. Kunze, K. Tanaka, Y. Uema, K. Kise, and M. Inami,“Smart eyewear for interaction and activity recognition,” in Proceedingsof the 33rd Annual ACM Conference Extended Abstracts on HumanFactors in Computing Systems. ACM, 2015, pp. 307–310.
[3] O. Amft, F. Wahl, S. Ishimaru, and K. Kunze, “Making regular eyeglassessmart,” IEEE Pervasive Computing, vol. 14, no. 3, pp. 32–43, July 2015.
[4] K. Masai, Y. Sugiura, M. Ogata, K. Kunze, M. Inami, and M. Sugimoto,“Facial expression recognition in daily life by embedded photo reflective
sensors on smart eyewear,” in Proceedings of the 21st International
Conference on Intelligent User Interfaces. ACM, 2016, pp. 317–326.
[5] A. Bulling, O. Cakmakci, K. Kunze, and J. M. Rehg, “Eyewear
computing–augmenting the human with head-mounted wearable assistants (dagstuhl seminar 16042),” in Dagstuhl Reports, vol. 6, no. 1.
Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2016.
[6] B. Tag, J. Shimizu, C. Zhang, K. Kunze, N. Ohta, and K. Sugiura,“In the eye of the beholder: The impact of frame rate on human eyeblink,” in Proceedings of the 2016 CHI Conference Extended Abstracts
on Human Factors in Computing Systems, ser. CHI EA ’16. NewYork, NY, USA: ACM, 2016, pp. 2321–2327. [Online]. Available:
http://doi.acm.org/10.1145/2851581.2892449[7] B. Tag, R. Mannschreck, K. Sugiura, G. Chernyshov, N. Ohta, andK. Kunze, “Facial thermography for attention tracking on smart eyewear:
An initial study,” in Proceedings of the 2017 CHI Conference ExtendedAbstracts on Human Factors in Computing Systems. ACM, 2017, pp.
2959–2966.
[8] K. Kunze, H. Kawaichi, K. Yoshimura, and K. Kise, “The wordometer–estimating the number of words read using document image retrieval andmobile eye tracking,” in Document Analysis and Recognition (ICDAR),
2013 12th International Conference on. IEEE, 2013, pp. 25–29.
[9] C. S. Ikehara and M. E. Crosby, “Assessing cognitive load with physiological sensors,” in Proceedings of the 38th Annual Hawaii International
Conference on System Sciences, Jan 2005, pp. 295a–295a.

43