F O R U M I N T E R A C T I O N T EC H N O L O G I E S
Envisioning, designing, and implementing the user interface require a comprehensive understandingof interaction technologies. In this forum we scout trends and discuss new technologies with the potential
to influence interaction design. — Albrecht Schmidt, Editor

Eyewear Computers for
Human-Computer Interaction

H

Andreas Bulling, Max Planck Institute for Informatics, Kai Kunze, Keio University
ead-worn displays
and eye trackers,
augmented andvirtual reality
glasses, egocentriccameras, and other“smart eyewear”
have recently emerged as a researchplatform in fields such as ubiquitous
computing, computer vision, andcognitive and social science. Whileearlier generations of devices were
too bulky to be worn regularly, recent
technological advances have madeeyewear unobtrusive and lightweight,
and therefore more suitable fordaily use. Given that many humansenses are located on the head, smart
eyewear provides opportunities
for types of interaction that were
impossible before now. In this article,
we highlight the potential of eyewear
computing for HCI, discuss available
input and output modalities, andsuggest the most promising future
directions for eyewear computing
research, namely multimodal usermodeling, lifelong learning, and largescale (collective) human-behavior
sensing and analysis.

HEAD-BASEDINPUT MODALITIESThe latest eyewear computers offer
a wide range of sensing and input
modalities, including eye tracking,egocentric cameras, microphones
and inertial sensors, and sensors formeasuring brain activity.
Head-worn eye tracking has seen
significant advances in recent years
and has therefore been a particularly
active area of research in eyewear
70

INTER ACTIONSM A Y –J U N E 2 0 16

computing. Eye movements are arich source of information about
the user, indicating their activities
and cognitive processes [1,2,3]. The
two most promising eye-tracking
methods for measurements in dailylife settings are Electrooculography(EOG) and computer-vision-based eye
tracking. EOG requires electrodes
positioned on the skin around theeyes and measures changes in theelectric potential fields caused by eye
movement. EOG is computationallylightweight and can therefore be
unobtrusively integrated into eye
glasses, such as J!NS MEME (Figure
1; jins-meme.com). However,
EOG can be subject to significant
signal drift and noise and provides
information only about relative eye
movements; it does not provide
accurate gaze direction.
In contrast, vision-based eye
tracking relies on special-purpose
eye cameras as well as infrared
Insights→→ As eyewear computers
become more lightweight andunobtrusive, they become moresuitable for everyday use.
→→ Given that many human senses
are located on the head,eyewear computing will enabletypes of interaction between
humans and machines neverbefore possible.
→→ Smart eyewear promises new
applications in multimodal usermodeling, lifelong learning,and collective human-behavior
sensing and analysis.

illumination, and provides gaze
direction in either 2D-scene-camera
or 3D-world coordinates. The scene
camera, readily integrated into
head-mounted eye trackers, records
high-resolution videos from theuser’s perspective and therefore
enables a variety of egocentric visionapplications. For example, the open
source PUPIL head-mounted eye
tracker from Pupil Labs (pupillabs.com/pupil) is affordable andfully customizable, and can be
implemented for long-term use(Figure 2). In addition, given thatthe eye cameras are typically highresolution, they can be used forcorneal imaging, which analyzes thescene reflected on the wet surface of
the user’s cornea (Figure 3). Corneal
imaging can provide contextual
information such as details on
the user’s attention, the objects
and interlocutors with whom theyinteract, and their surroundings.
Taken together, head-worn eye
tracking has significant potential
to enable a new class of mobile HCI
applications, in particular whencombined with other smart eyewear
such as head-mounted displays(HMDs).
Egocentric vision has seen asimilar increase in research activity.
Egocentric vision can provide
additional user context, for example,
the objects users are interactingwith as well as their activities andsocial interactions [4]. It can also be
used for direct interactions such asdetecting hand gestures, as well as forsecurity applications such as secure
identification, document verification,
INTER ACTIONS. ACM.ORG

Figure 1. From fNIRS and mobile eye tracking glasses to Google Glass to unobtrusive smart glasses (J!NS MEME prototype recording eye andhead movements).

and content hiding [5] (see framework
in Figure 4). Embedded computers
have reached a level of computing
power and miniaturization thatmake it possible to use manystate-of-the-art computer-visionalgorithms on head-worn devices,such as those for activity, gesture,and text recognition. However, manyalgorithms need to be adapted oreven rewritten from scratch because
egocentric vision is fundamentallydifferent from established computervision setups. In particular,
egocentric cameras move with theusers and therefore are not only
subject to blur but also observe theworld from viewpoints for which little
training data is available.
Given they are relatively cheap,
small sensors, microphones canbe easily integrated into eyewear
computers. Microphones havebeen used for tasks such as speech
recognition or, when combined withbone-conduction speakers, biometricFigure 2. PUPIL from Pupil Labs is anaccessible, affordable, and extensible open
source platform for head-mounted eye
tracking, mobile gaze-based interaction,and egocentric vision research.

Figure 3. Current head-mounted eye trackers
feature high-resolution eye cameras that canbe used for corneal-imaging applications,such as gaze estimation, hand-gesture recognition, or semantic scene analysis.

user identification [6]. Ambient
sound in particular is a rich sourceof contextual information about theuser’s environment and the people
and objects with which they areinteracting. Voice input is also a
viable interaction modality for headmounted computers, as seen in Google
Glass. Yet, when used in public, voiceposes privacy and social challenges
that are interesting from an HCI
research perspective.
Recent work has investigated theuse of other lightweight sensors. For
example, many projects have focusedon physical-activity recognition usinginertial motion sensors, including
sensors attached to the head [3].

Please enter
this PIN:
1234

Please scan
header tag

Please scan
block #

Successfullyauthenticated(a)Successfully
validated(b)

Please scan
tag #

(c)

Figure 4. The Ubic framework combines a head-mounted display, such as Google Glass, with egocentric vision methods to bridge the gap between
digital cryptography and the physical world. The framework covers key cryptographic primitives, such as secure user identification, document
verification using a special secure physical document format, and content hiding.
INTER ACTIONS. ACM.ORG

M A Y –J U N E 2 0 16

INTER ACTIONS

71

F O R U M I N T E R A C T I O N T EC H N O L O G I E S

Figure 5. Affective Wear detecting facial expressions [7].

Similar to other modalities, headworn sensors can provide additional
information about the wearer, suchas their cognitive tasks and attention[1,2]. Head motion can also indicatesocial activities and be used as an
input modality for HMDs. Katsutoshi
et al. investigated embedded photoreflective sensors to recognize awearer’s facial expressions in daily
life [7]. The system leveraged the skindeformations created when wearers
make facial expressions by measuring
the proximity between skin surface
and glasses frame (Figure 5).
Finally, brain activity isparticularly interesting in the context
of head-worn HCI because it can
provide additional insights into users’
cognitive processes and can be used inbrain-computer interfaces. The
two sensing modalities mostpromising for wearable
measurement of brain activity areelectroencephalography (EEG)and functional near-infrared
spectroscopy (f NIRS). Recent devices
such as the Emotiv Insight headset(emotivinsight.com) or the Hitachi
HOT-1000, a consumer f NIRS device,point the way toward unobtrusive
brain-activity sensing as well as
mobile brain-computer interfaces.
Zeiss

Lumus OE 40

HEAD-WORNOUTPUT MODALITIESOur impressions of the world andour memory of it are constructedin large part by the human visualsystem. There are more than 30
areas concerned with vision in thebrain, which occupy more thanhalf of the total surface area of thecortex [8]. Head-worn displays are
therefore positioned to play a key rolein potentially extending our visual
sense by overlaying the wavelengths
our eyes do not naturally see (suchas infrared), potentially overlaying
zoom-lens output, and overlaying 2D
or 3D computer-generated imagery.
Optical design of head-worn displays
involves a rich space of design
parameters and trade-offs (see Figure
6 for some example displays). Main
modes of operation include opticalsee-through, video see-through,and opaque displays. Optics canbe designed for one eye or both.
Important optical design parameters
include eyebox, eye clearance, fieldof view, virtual-image location anddistance, wavelength band, and image
quality. For augmented reality (AR)applications, a see-through display
has several advantages, including
minimal obstructions of the visual

field, making them safer to wear. In
addition, the resolution, distortion,and depth perception of the real scene
can be as good as that which is seen
with the naked eye. In the consumer
space, aesthetics and fashion makeup the single most important design
parameter.
At the moment, there is no publicly
known optical architecture that cansatisfy all of the desired constraints(see-through, aesthetics and fashion,
full color, large field of view, and soon) simultaneously [9]. On the positiveside, the increasing popularity ofsmart watches indicates that there
are various microinteraction-based
AR applications that can be realizedwith a small field of view. There aresmart watches in the market withapproximately 30 mm 2 displays,
which, when viewed at 12 inches,would correspond roughly to a 6x6degree field of view. One challenge isthat the display has no access to theuser’s exact view as rendered on theretina. Corneal-feedback augmented
reality is a novel technology with thepotential to address this problem byproviding a closed-feedback loop bycontinuously inspecting the reflectionsin the eye. More futuristic are contactlens-based displays. A main drawback
is the more invasive nature of thisapproach from a user-acceptance pointof view, especially for users who do notneed corrective lenses.
Other feedback modalities
potentially available on eyewear
computers are sound, tactile, andsmell. Sound has long been used toprovide information as a secondary andreinforcement channel, when the userneeds to focus visually on the task at
hand. More interesting, however, arenewer research works that explore how
to alter what we hear. This even led tosome pioneering consumer products
such as Active Listening by Doppler
Labs (https://www.dopplerlabs.com).

Microsoft Hololens

Google Glass

Figure 6. Different head-up displays.

72

INTER ACTIONSM A Y –J U N E 2 0 16

INTER ACTIONS. ACM.ORG

In this case, sound is not used as mere
notification; rather, the devices alter
the sound perception to filter given
frequencies to be able to hear voices
more clearly, for example. Sonic
interface design in AR environments
seems to be a promising new research
direction, especially when combinedwith other eyewear sensing andfeedback modalities.
Research in tactile feedback
so far has focused on people withdisabilities (mostly people who are
vision- and hearing-impaired). A
major advantage of using haptics
over sound: It is privacy conserving.
Haptics on the head for non-disabled
people is not very well explored. Early
work investigates haptic feedback for
navigation on smart glasses combining
eye gaze and tactile output [10].
Smell as feedback might seem
unconventional at first. Yet there
are interesting links between memory
retrieval processes and smell.
A specific odor might enable a user
to easier recall specific facts. And,especially on eyewear computers,
it is easy to implement them in apersonalized fashion (so that otherpeople aren’t disturbed).

WHAT IS DOWN THE ROAD?

The input and output modalities
available on current smart eyewear
already enable a wide range of mobile
applications and, at the same time,pose interesting challenges forhuman-computer interaction anduser interface design. Smart eyewear
also points toward completely
new applications, the three mostpromising of which are multimodal
user modeling, lifelong learning, andlarge-scale (collective) human-behavior
sensing and analysis.
While current smart eyewear
typically relies on a single input andoutput modality, future head-worndevices can exploit multiple modalities
at the same time. Multimodal sensing
will enable researchers to model user
behavior not only more holistically but
also more robustly and with improved
performance compared with whenusing only a single modality [1,2,3].
Multimodal user modeling requires us
to establish which modalities are best

suited for a given task, and in addition,these modalities need to be combinedin the right way to allow for seamlesssensing and natural interactions anduser feedback.
With further advances inminiaturization and improvements
in signal processing, egocentric
vision methods, and battery
lifetime, future smart eyewear has
significant potential for long-term
sensing, modeling, and analysis ofuser behavior. Current systems canmonitor users only over relatively
short amounts of time and therefore
have a temporally limited view on
their behavior. Methods for lifelong
learning, however, can leverage thefull information content available inall interactions that users perform
explicitly or implicitly with computing
systems and with each other over theyears. Lifelong learning therefore
represents both a paradigm shift
and key characteristics to realize
the vision of collaborative or evensymbiotic human-machine systems.
Finally, while all of thesedevelopments will be beneficial for
individual users, eyewear computing
promises multimodal user-behavior
modeling and lifelong learning atlarge, in other words, for collectivesof large numbers of users in daily life.
Mobile phones are already starting
to be employed for collective humanbehavior analysis, for example toanalyze movements of large groupsof people at large-scale events
such as music festivals. Eyewear
computers show even bigger promise
for such analyses, given their close
proximity to the user and theirlocation on the head.
Given the increasing numberof consumer devices and recent
advances in head-mounted eye
tracking, egocentric vision, as wellas head-mounted augmented andvirtual reality displays, it is a perfecttime for HCI researchers and userinterface designers to take up both theopportunities and the challenges ofsmart eyewear.

ACKNOWLEDGMENTS

This article is based, in part, on
discussions with participants of the

Dagstuhl Seminar 16042 “EyeWear
Computing - Augmenting the Humanwith Head-Mounted Wearable
Assistants” (http://www.dagstuhl.
de/16042).
Endnotes
1. Amft, O. et al. Making regular eyeglasses
smart. IEEE Pervasive Computing 14, 3(2015), 32–43, 2015.
2. Bulling, A. and Zander, T.O. Cognitionaware computing. IEEE Pervasive
Computing 13, 3 (2014), 80–83, 2014;
http://dx.doi.org/10.1109/mprv.2014.42
3. Ishimaru, S. et al. In the blink of aneye: Combining head motion andeye blink frequency for activity
recognition with google glass. Proc. ofAugmented Human 2014; http://dx.doi.
org/10.1145/2582051.2582066
4. Fathi, A., Farhadi, A., and Rehg, J.M.
Understanding egocentric activities. Proc.
of ICCV 2011.
5. Simkin, M. et al. Ubic: Bridging the gap
between digital cryptography and thephysical world. Proc. of ESORICS 2014,
56–75; http://dx.doi.org/10.1007/978-3319-11203-9_4
6. Schneegass, S. et al. SkullConduct:
Biometric user identification on eyewear
computers using bone conduction throughthe skull. Proc. of CHI 2016; http://dx.doi.
org/10.1145/2858036.2858152
7. Katsutoshi, M. et al. Facial expression
recognition in daily life by embedded
photo reflective sensors on smart eyewear.
Proc. of IUI 2016.
8. Kandel, E. et al. Essentials of Neural Scienceand Behavior. McGraw Hill Professional,
1995.
9. Cakmakci, O. and Rolland, J. Head-worndisplays: a review. Journal of Display
Technology 2, 3 (2006), 199–216.
10. Kangas, J. et al. Using gaze gestureswith haptic feedback on glasses. Proc. of
NordiCHI 2014.
Andreas Bulling is head of the Perceptual
User Interfaces Group at the Max Planck
Institute for Informatics as well as the Clusterof Excellence on Multimodal Computing andInteraction at Saarland University, Germany.
His research interests are in human-centeredcomputing, human-computer interaction, andcomputer vision.
→→ bulling@mpi-inf.mpg.de
Kai Kunze is an associate professor at
the Graduate School of Media Design, Keio
University, Yokohama, Japan. His researchinterests include eyewear computing,human-computer interaction, and wearable
computing with a focus on amplifying skills andaugmenting the embodied mind.
→→ kai@kmd.keio.ac.jp

DOI: 10.1145/2912886 COPYRIGHT HELD BY AUTHORS. PUBLICATION RIGHTS LICENSED TO ACM. $15.00
INTER ACTIONS. ACM.ORG

M A Y –J U N E 2 0 16

INTER ACTIONS

73