D i g i t a l l y E n h a n c ed R e a l i t y

Making Regular
Eyeglasses Smart
Software apps running on smart eyeglasses offer unobtrusive universal
assistance systems that can support wearers in their daily lives. The
authors describe a blueprint of the embedded architecture of smart
eyeglasses and discuss findings from using smart eyeglass prototypes.

M

ost of our senses, vital signs,
and actions involve the head,making the human skull oneof the most interesting body
locations for the simultaneous sensing and interaction of assistanceapplications. Although hearing aids and mobile
headsets have become widely accepted as headworn devices, users in public spaces often consider novel head-attached sensors and devices tobe uncomfortable, irritating,or stigmatizing. Nevertheless,numerous wearable computing
Oliver Amft and Florian Wahl
studies have shown that headUniversity of Passau
worn sensors and adequate
Shoya Ishimaru
signal processing could reveal
cognition-related behavior andOsaka Prefecture University
essential vital parameters (suchKai Kunze
as heartbeat and breathing
Keio University
rates). Behavior and vital datais the key component for many
assistance applications in daily
life, from those that offer memory augmentation
to those advising chronically ill patients. Assistance applications thus require continuous measurements and context estimation.
Today’s regular eyeglasses could fill the gap
between sensing and assistance in daily life.
Most eyeglasses are publicly accepted accessories, often worn continuously throughout theday, rendering them an ideal platform for additional assistive functions. In contrast to Google
Glass and early wearable system approaches
that just attached devices to standard glasses,

2

PER VA SI V E computing 

we envision exploiting smart eyeglasses for assistance applications through fully integrated
sensing, processing, and interaction functionsin the regular spectacles design, thus maximizing unobtrusiveness. We also envision embedded software apps that let wearers dynamicallyalter assistance functionality depending on momentary needs. With a rich set of software appsthat can be selectively run on smart eyeglasses,
one pair of smart eyeglasses could serve various purposes in daily life and provide novel assistance applications in continuous monitoring,augmentation, and therapy, beyond what smartphones and smart watches can achieve today.
They would offer much more than short-term
interaction and quick information access.
Although the first steps toward commercial
smart eyeglasses are currently being made (with
J!NS MEME; www.jins-jp.com/jinsmeme/en),scientific analyses and studies of variable assistive functions in smart eyeglasses are lacking.
Consequently, opportunities and requirements
for smart eyeglasses and assistance applications are unclear. Following from commercial
examples, smart eyeglasses seem to be primarily useful in sensing and processing tasks, giventhat eyeglasses provide access to unique sensor locations near the head. Here, we present
an architecture for integrating technology into
traditional eyeglass designs, discuss ergonomic
design requirements, and derive recommendations for further smart eyeglasses developments.
In three case studies, we observed the potentialof using smart eyeglasses for assistive functions.

Published by the IEEE CS n 1536-1268/15/$31.00 © 2015 IEEE

(a)

(b)

(c)

Figure 1. Sensing in eyeglasses varies from (a) attaching devices to glasses (Google Glass) to initial attempts at integrated smart
eyeglasses, such as the (b) J!NS MEME electrooculography (EOG) glasses, which measure eye movements and head motion,paired with several applications. (c) EnWake eyeglasses provide blue light therapy to help wearers manage their sleep-wake cycle,
providing visualizations using a smartphone. (The EnWake images are courtesy of EnWake Holding BV; used with permission.)

Sensing in Everyday
Eyeglasses
Humans receive approximately 87 percent of all sensory input via the head,1
spurring the recent development of interaction and notification approaches involving head-mounted displays (see Figure 1). For example, Google Glass builds
on the concept of micro-interactions totimely accomplish information search
and feedback lasting only a few seconds(see Figure 1a). In contrast, many assistance applications require continuous
sensing, processing, and interaction.
For such continuous operations,clinical measurement practices indicatethat the head is a good place to sense
and monitor body functions. For example, blood oxygen saturation, heartrate, and respiration are measured at
the ear lobe. Core body temperature
can be estimated from head-attachedsensors. Similarly, human behavior assessments—related to diet, stress, concentration levels, and so on—often involve direct or indirect ­measurements

July–september 2015

taken near the head. For example,
acoustic sensors and strain gauges havebeen used to monitor chewing andswallowing, both of which can indicate
dieting habits and stress levels. 2 Pupil
dilation and blink rate have been used
to estimate concentration.3 Abnormal
walking patterns, which can indicate
osteoarthritis and past knee- or hip-replacement surgery, were identified fromhead-worn motion sensors.4 Recently
proposed devices incorporate biopotential electrodes and acceleration sensors
to support wellness and sports applications that encourage meditation andrelaxation or detect the user’s heart
rate or count his or her steps. Examples of these devices include i­ntelligent­headbands, such as Muse (www.choosemuse.com) and Melon (www.thinkmelon.com), and sensor earplugs, suchas those from PEARSports (www.pearsports.com). Although none of theseapproaches use eyeglasses, and most
focus on individual applications, their
sensing, processing, and interaction

components are compatible with ourarchitecture for smart eyeglasses.
When it comes to exploiting glasses
for augmented and virtual reality applications, however, there has been
substantial research. Such work has
focused on improving see-through orlook-around optics for information
display—examples include the Oculus
Rift (www.oculus.com/rift) and Sonysmart glasses (https://developer.sony.
com/develop/wearables/smarteyeglasssdk). Furthermore, Oakley’s sunglasses
with an embedded music player andheadphones illustrate the wider market potential of technology in eyewear(http://en.wikipedia.org/wiki/Oakley_
THUMP). However, bulky frame-attached processing devices or otherwise
cumbersome layouts make many existing eyewear designs impractical foreveryday use. Technology miniaturization and power optimization will eventually make augmented reality functions smaller, but even today, assistive
smart eyeglasses could be realized.

PER VA SI V E computing

3

Digitally Enhanced Reality

P5

P5

P6

P6

P3

P3
Legend on interaction locations

Legend on sensor/system locations

P2

P4

P7

P7

P4

Environmental sensors

Touch & gesture interaction

Physiology sensors(potential skin contact)

Visual interaction(projected on glass or in retina)

Contactless & visual sensors

Acoustic & haptic feedback

Activity & location sensors,
electronics, communication& power supply

P2

P1

P1

(a)

(b)
P1: Temple end

P2: Temple ear bend

P3: Temple front

P4: Temple hinge

P5: Outer rim

P6: Nose pad

P7: Pad arm

Figure 2. An illustration of smart eyeglasses architecture and device positioning: (a) sensor/system locations and (b) interaction
locations. Positions on eyeglasses: temple ends/earpiece (P1); ear bends, which can be used as support over the ears (P2); temple
sides and fronts, near the visible region of temples above the cheek (P3); the hinge connecting the lens frame and temple end
piece (“butt-strap”) (P4); the frame rim, which carries the lenses (P5); the nose pads (P6); and the pad arm/bar, connecting
symmetric frame parts (P7).

Toward everyday settings, few attempts have been made to embed technology directly into the standard eyeglasses form factor. In addition to a
motion sensor, the J!NS MEME (seeFigure 1b) integrates electrodes in thenose pads of a traditional eyeglasses
form factor. Electronics are placed behind the ears in the temple ends (seeFigure 2a for an illustration of this andother terms related to eyewear frames).
Nose-pad, pad-arm, and temple electrodes are used to capture biopotential, used in electrooculography (EOG)
analysis to determine eye movement.
In combination with head motion, theeyeglasses aim to detect tiredness, concentration, sleepiness, and energy expenditure during physical activity. The
Optalert Fatigue Management Glasses
use an LED and a photodetector to4

PER VA SI V E computing 

monitor drowsiness during driving(http://optalert.com). Optalert monitors the eyelid and derives amplitudevelocity rates of blinks. The EnWake
eyeglasses (see Figure 1c) target lighttherapy and feature blue light LEDs
diffused by a prism to stimulate the human circadian system through the eyes
(http://ece.nl/enwake-minimizing-jetlags). Although EnWake eyeglasses areintended for noncontinuous stimulation only, their approach indicates potential for care and therapy applications
integrated into eyeglasses.
Whereas Optalert and EnWake were
designed for one function only, the
J!NS MEME glasses illustrate the trend
toward integrated multipurpose smart
eyeglasses that could be used in different monitoring and assistance applications in everyday life. Beyond current

eyeglasses designs, we envision manyother assistive functions on smart eyeglasses—from software apps runningon smart eyeglasses to functions exploiting dedicated hardware.

Smart Eyeglasses
Architecture
A smart eyeglass design essentially depends on the form factor of the supporting eyeglass frame. Sensing andinteraction modalities, the positioningof functions on eyeglasses, as well as
the integration of processing electronics and powering are determined by the
supporting frame (see Table 1 for more
information).
Ergonomic Design
Smart eyeglasses provide a unique approach to ergonomic sensor ­placementwww.computer.org/pervasiveTable 1. An overview of smart eyeglass components and integration options. The location indicators (P1–P7)
are illustrated in Figure 2.
Component

Possible location

Device examples

Integration considerations

Environmental
sensors

Temple fronts (P3), temple hinges (P4), outerrims (P5), and pad arms (P7), because they
are most exposed to the outside world

Devices that monitor­environmental temperatures, humidity, sounds,
and so on

The sensors weigh only a few
grams Sensor appearance should­minimize obtrusivenessPhysiologicalsensors

Nose pads (P6), pad arms (P7), and temple
ear bends (P2), so the sensors can maintain
constant contact with the skin and controlledcontact pressure (given by the weight of theeyeglasses)
For sports glasses, the skin-contacted regionsof the inner rims and temple fronts

Devices that monitor skintemperature or that measure eye movement or heart
activity (for example, using
biopotential electrodes)

The electrode weight is typicallynegligible, and the size can be
adjusted to eyeglass dimensions fornose pads and temples

Contactless andvisual sensors

Inner rims (P5) and the inner side of a padarm (P7), so the sensors can directly measure
the face and eyes to monitor blinks or pupils

Low resolution cameras or
LED-based detectors

The weight and size are critical(especially for optical sensors)
Weight balance can be reachedwhen placing components thatweigh more than ~10 g at the
pad arm (P7) rather than temple
edges (P4)

Activity andlocation sensors

Temple ends (P1) provide a space for sensors
that do not require exposed position or skincontact

Orientation sensors or GPS

Temple ends (P1) are least constrained in weight, size, and weightbalance
Sensors can be integrated into themain electronics

Touch interaction

Temple fronts (P3) are well suited for touch
micro-interaction (for less than two seconds)
and can support multiple touch commands
along the temples
The pad arm (P7) can be used for abutton-like interface

Touch pads

Touch sensors are planar and weighin the milligram range

Visual interaction

Temple fronts (P3) and rims (P5) can carry
optical feedback or assistive devices

Projection optics or LEDs

Optical devices can be large
and heavy (25 g or more), and­single-eye optical devices should
be avoided due to uneven weightbalanceAcoustic andhaptic interaction

Temple ear bends (P2), optionally with
attached headphones.

Headphones or buzzers

Additional weight (∼ 10g) can besupported at the rear side of temples (P1 and P2)

Electronics, communication, andpower supply

Temple ends (P1) are the best-suited location,with temple fronts (P3) as an alternative
Electronics, communication, and the power
supply (battery) can be distributed onto both
temples and interconnected via the temples
and rim

Microcontrollers, wirelessmodules, antennas, batteries, and so on

Although size and weight at temple
ends are the least constrained,
weight balance must be achieved
between both temples

at important head locations. Nevertheless, key ergonomic design constraints
including size, weight, and weightbalance around the pad arm must be
respected. We consider that the basic
shape and dimensions of spectacles
should be maintained for smart eyeglasses, which limits component selection. In particular, for many assistance applications, smart eyeglasses
will likely integrate s­ensing, data

July–september 2015

­ rocessing, and communication funcp
tions in addition to the power supply.
Interaction functions must be used selectively, due to the displays’ relatively
large size and weight requirements.
Finally, imbalances—due to devices
attached at one temple side, for example—result in wearer discomfort andirritation. In contrast, eyeglasses can
carry balanced weight at the temple
ends. For example, the main electronics

and battery can be placed on respective
temple ends to balance weight.
Size
Our comparison of dimensions among
regular eyeglasses showed that the distance between the temple hinges andskin, as well as rim size, are importantmeasures. Smart eyeglasses thus should
use rim and temple designs similar toregular eyeglasses. In contrast, thePER VA SI V E computing

5

Digitally Enhanced Realityv­ ertical size of temple sides and ends
is less limited, as J!NS MEME and EnWake illustrate.
WeightFor everyday continuous use, weight appears to be a critical constraint. In a user
study on spectacles, participants rated
the total weight, giving eyeglasses thatweighed 100 grams an average score of
8 out of 10 points, where 10 indicatedhigh wearing comfort.5 For eyeglasses
that weighed 75g, the average score improved to 9 out of 10 points. Frame andsight-correcting plastic lenses typicallyrequire approximately 50g, thus leaving
a practically useful weight budget of 25
to 50g for additional smart eyeglasses
components. Our personal discussions
with J!NS researchers confirmed thisweight budget.
Power Consumption
Runtime requirements of smart eyeglasses will be similar to smart watches
and smartphones today—that is, smart
eyeglasses need to operate for at least
one day without needing to be recharged. The temple ends support a rechargeable battery with approximately
250 mAh total capacity, similar to batteries in a small smart watch.
Many sensors considered in our architecture are available in low-power designs, and related processing strategies
are well documented. For example, inertial sensors and Photoplethysmography(PPG) sensors could be continuously
operated, because only few milliamperes are needed to sample and process
sensor data. N­ evertheless, components
including cameras, GPS, and displaysfor visual interaction typically require
several 10 mA or more. ­High-powercomponents thus are not well suited forcontinuous operation and need appropriate sampling strategies.
Skin Contact
Sensors in contact with the skin are
central for many behavior and physiological monitoring tasks, including EOG and heart rate. Eyeglasses

6

PER VA SI V E computing e­ rgonomics provide useful spots forskin contact, such as nose pads, thepad arm, and temple ear bends (seeFigure 2). In addition, tight-fitting
sports glasses could use temple fronts
for sensors that require skin contact(at P3 in Figure 2). Most skin contact locations for eyeglasses provide
constant contact pressure due to theeyeglass weight, supporting reliablemeasurements.
Placement Invariability
Eyeglass designs support correct placement on the head, and large misalignment is less likely than with other wearable devices. However, eyeglasses tend
to be repositioned during daylong wear,
and for smart eyeglasses that include
sensors and interaction functions, repositioning and motion-intensive activity
could lead to sensor and input artifacts
that require corrective processing. In
motion-intensive applications, specifictight-fitting sports-eyeglass designs
might be needed to reduce artifacts.
Component Priorities
Several locations could be used for different components, such as temples,
hinge, rims, and pad arms, providing
flexibility for designers. Placement conflicts are unlikely for most environmental, activity, and location sensors, because their placement does not dependon an exact position and the components are small. In contrast, physiological sensors are usually location-dependent—for example, EOG electrodes
on nose pads (P6). Similarly, many­interaction modalities will require specific locations, such as acoustics at P2
or touch at P3. Due to the relevance ofsensing in assistance applications, interaction in smart eyeglasses might thus
provide limited functionality or could
be omitted. Instead, external interaction options, such as a Web portal, display, or smartphone, might be used.

Apps for Smart Eyeglasses
Among the wide range of potential
software apps that can be executed on

smart eyeglasses, we investigated examples that leverage earlier work using
head-attached assistive technology (seeTable 2).4,6–16 For all apps, smart eyeglasses require programmable processing and communication components
to provide processed information to
feedback services (such as Web portals)
or to interact with surrounding devices(such as smartphones). Consequently,
running smart eyeglass apps requires
only that app-specific sensors and interaction modalities be available.
Application domains of previous
head-attached systems span a wide
range of fields, from health and fitness
to tourism to interaction and cognitive
aids. Nevertheless, the required sensor
and interaction modalities overlap, suggesting that smart eyeglasses could indeed serve multiple purposes.
App Clusters
Here, we group apps into clusters thatuse common smart eyeglasses hardware and cover related assistance applications (see Table 2).
Health and interaction. This cluster addresses everyday apps in health monitoring, the quantified self, and interaction. Activity and physiologicalinformation is processed and communicated to external feedback devices.
Smart eyeglasses might provide limited notification interfaces, such as indicator LEDs. All functions have low
size, weight, and power requirements
and thus could operate continuously
throughout the day.
Location and cognition. This cluster emphasizes location tracking and remembering applications, with hardware
requirements similar to the health andinteraction cluster. We view this as aseparate cluster, because cameras canbe considered intrusive in public settings and raise power requirements.
Gaze control. The apps in this cluster are
suitable for touchless interaction andhands-free control using eye ­gestures.

www.computer.org/pervasiveTable 2. Potential software apps that can be executed on smart eyeglasses to assist in different situations. Apps were clusteredaccording to common smart eyeglasses hardware requirements. The location indicators (P1–P7) are illustrated in Figure 2.
Hardware configuration

Smart eyeglass apps and reference to details

Sensors, interaction, and location

Health and interaction

Energy expenditure estimation6

Acceleration sensor (P1), light sensor (P7),heart rate and respiration rate via Photoplethysmography (PPG) at the ear (P1 and P2)
Indicator LED in a visual space (P5) Additional
feedback/interaction via Web portal, computer/tablet, and smartphone

Location and cognition

Locating in indoor environments, such as offices9
Locating via pictured points of interests in unknown
environments10
Remembering people from camera images11

Inertial sensors (P1)
Laser range sensors and camera in viewing
direction (P3, P4)
Feedback/interaction via Web portal andsmartphone

Gaze interaction

Detecting eye gestures relative to the head during­reading12 for touchless display control
Eye gaze as hands-free control of a music player13
Quantifying reading habits (in this work)

Inertial sensors (P1)Electrooculography (EOG) electrodes aroundeyes (P5 and P6)
Interaction via a controllable device, such asa display

Therapy

Acoustic monitoring of chewing microstructure­parameters and food consumption patterns14

Microphone at ear (P2)
Feedback/interaction via Web portal and­smartphone

Cognitive aid
and control

General memory augmentation15
Monitoring driver vigilance16
Cognitive workload monitoring to pace learning(in this work)

Camera and optical eye tracking in visual
space (P5)
EOG electrodes around eyes (P5 and P6)­Interaction via feedback device, such as a car
dashboard, display, or computer

Detecting gait abnormalities4
Monitoring heart activity and respiration7
Managing subtle/private notifications via indicator LED8
Monitoring light exposure to adjust the sleep-wakecycle (in this work)

Output is realized via external devices,such as displays. Eye-based interaction requires specific sensors (such as
EOG). As J!NS MEME demonstrates, acombination of a few biopotential electrodes and inertial sensors is sufficientfor monitoring tiredness, concentration, sleepiness, or energy expenditure.
EOG sensors add to the size, weight,
and power requirements, but they canbe operated continuously throughout
the day.
Therapy. This cluster exemplifies alifestyle therapy application using
application-dedicated sensors andactuators. Here, dietary advice isgenerated based on acoustic data processed on the smart eyeglasses. Other
therapy applications might requiredifferent dedicated sensors or actuators (such as blue LEDs for EnWake).
Cumulative feedback and interaction
options, such as goal setting, could be
realized via Web portals and smartphones. Even small wearable systems
can process continuous audio data

July–september 2015

and potentially forward intermediate information to a cloud service,thus providing monitoring throughout the day.
Cognitive aid and control. This cluster
uses an eye-facing camera, optical eyetracking, and EOG sensors to analyze aperceived or actual cognitive load. Dueto the complex monitoring components(eye-facing camera), smart eyeglasses inthis cluster might serve in focused control tasks and cognitive assistance applications, such as for cognitive training apps.

Design Considerations
In our architecture and ergonomic design considerations, we found that integrating sensing, data processing, andcommunication functions is essential
for smart eyeglasses. However, eyeglasses-integrated interaction is often
limited to simple indicators, such as
an LED for subtle notifications, due tosize, weight, and power consumption
requirements, and none of the apps

envisioned in Table 2 offer a glassesattached display. Moreover, none of
the commercial smart eyeglasses (J!NS
MEME, EnWake, or Optalert) provide interaction functions. Althoughinteraction is essential for assistanceapplications, notifications could beperceived as highly disruptive in smart
eyeglasses. Similarly, due to resource
limitations, input modalities often only
provide micro-indications.
Our analysis, however, indicates thatvarious external interaction options canbe used with smart eyeglasses in assistance applications, such as Web portals,
external displays, and smartphones.

Case Studies
We highlight three promising assistive
applications, where smart eyeglasses
provide cognitive and health assistancefunctions. To track basic cognitive
functions, we explored recognizing the
cognitive load; to support learning, wequantified reading habits; and to trackhealth and circadian rhythm, we monitored light exposure behavior.

PER VA SI V E computing

7

Digitally Enhanced Reality

Recognizing Cognitive Workload
Real-time tracking of changes in cognitive workload while users performtasks could lead to a paradigm shift in
education, because we could give learners challenging problems without asking too much from them. Workload estimation can help judge the likelihoodof a user successfully completing a task.
Current measurement practice. From
psychology research, a correlation between eye motion features and cognitive workload is well known.3 Yet moststudies still use questionnaires to detect
workload level and focus on basic experimental setups using medical monitoring equipment. Functional near-infrared
spectroscopy (NIRS) is often considered
as a reference for detecting changes inbrain activation related to cognitive
workload. NIRS measures changes inoxyhemoglobin in a particular brain
area. However, NIRS requires wearing aheadband or similar bulky device. Basedon earlier work,3 we are confident thatsmart eyeglasses could estimate both
perceived and actual cognitive workload
using blink frequency and pupil diameter measurements, respectively.
Smart eyeglasses use cases. A smart eyeglasses app can easily assess blink frequency and estimates perceived workload using two-electrode EOG sensors
or an infrared distance measurement
integrated in frames and the nose pad(P5 and P6 in Figure 2). In contrast,
an app running on smart eyeglasses toestimate actual workload must determine pupil diameter and track ­pupils.
­B ecause ­pupil tracking typically requires an eye-facing camera, the battery runtime is constrained to few
hours. Due to camera size, smart eyeglasses specifically intended for reading
and actual workload estimation (“reading glasses”) are conceivable.
Learning tasks are usually displayedon a computing system, such as a tablet for grammar learning. The smart­eyeglasses app could thus forward
workload estimates to the display,

8

PER VA SI V E computing 

where the optimal difficulty level could
be continuously adjusted.
Another use case is tracking cognitive workload over a day—for air-traffic
controllers, for example. A smart eyeglasses app could connect to a computeror smartphone, so when a user loses focus (as evidenced by a detected decline
in cognitive load), warnings could be
displayed or the interface could change
to recapture the user’s attention.
Initial analysis. Using NIRS as a refer-ence, we investigated the relationshipbetween cognitive load and eye motion using features from EOG glasses.
Eight participants (three females andfive males, between ages 19 and 32)
performed a dual n-back memory task(n = 1, 2, 3, 4), where the difficulty level
increases with n. We used alphabet letters as acoustic stimuli and single highlighted squares in a grid of eight squares
as visual stimuli. The n-back task iswell explored using NIRS and aims at
a linear increase in brain activation forincreasing difficulty level. Participants
performed each difficulty level for one
minute with eight repetitions, restingfor one minute between repetitions. We
interviewed participants after each repetition to determine if they were “giving up” prematurely during the game.
Assigned tasks were performed using a
Latin square method to swap task typeand difficulty level.
We recorded data from a stationary
eye-tracker (SMI RED) at 250 Hz, a
42-channel NIRS device (Shimadzu
LabNIRS) at 6 Hz, and five-point EOG
eyeglasses (J!NS MEME) at 100 Hz.
Figure 3 illustrates the experimental
setup. We applied all 42 NIRS channels to the prefrontal cortex responsible
for most complex planning, learning,decision making, and action during
demanding cognitive tasks. For our
NIRS analysis, we applied a standard
methodology to assess brain activation
by selecting the NIRS channel with thehighest activation and then we averagedchanges in this channel. We measured
pupil diameter using the eye tracker

and changes in blink frequency using
the J!NS MEME EOG signal.
Figure 3 shows that with increasingtask difficulty, change in pupil diameterincreases and blink frequency decreases.
We confirmed task difficulty in participant interviews and by analyzing NIRS
brain activation. Some task recordings
from higher difficulty levels were notavailable, because participants gave up
or could no longer perform the task.
We identified “giving up” as drop in theoverall NIRS brain activation and as adrop in the participant’s performance(getting no or very few correct answers
compared to other participants). The“giving up” state was also confirmedin participant interviews. Our ANOVA
tests confirmed a relationship between
the task difficulty as an independent
variable and the change in pupil diameter, p < 0.05, with F = 6.89, Fcrit(3, 4)= 6.59 (where denotes the F-distribution
cumulative distribution function); andnormalized blink frequency p < 0.05, F= 6.63, Fcrit(3, 4) = 6.59, both separately
considered as dependent variables. The
Pearson product-moment correlation revealed good positive correlation for taskdifficulty and change in pupil diameter (r= 0.71), and decent negative correlation
for task difficulty and normalized blink
frequency (r = –0.53).
The results have yet to be verified,as the overall blink frequency and thepupil diameter are crude measurements
and we used only a small user sample.
Greg J. Siegle and his colleagues showthat the relation of blink frequency
and pupil diameter over time could
provide more insights regarding cognitive workload.3 Analyzing the time series of brain activity, blinks, and pupilchanges is a promising direction, where
smart eyeglasses might provide longterm assistance.
Quantifying Reading Habits
Increasing reading volume is in linewith cognitive merits, including
larger vocabulary and higher general­knowledge.17 Reading is entertaining
and has social value too. Higher readingwww.computer.org/pervasivev­ olumes in adolescents are correlatedwith higher self-esteem and improved
cognitive and emotional well-being.18
Current measurement practice. Most
work focuses on reading detection
only.12 A few online reading services,such as Goodreads (www.goodreads.
com), enable readers to track their habits manually. So far, automatic reading
habit recognition and recording has notbeen implemented or used to improvereading habits.

Initial analysis. Initially, we used mobile
optical eye tracking (see Figure 4a) toimplement a word count system and­detected reading using simple saccadefeatures (average saccade length, direc-

July–september 2015

0.9

5.5
Change in pupil diameter

0.8
Blink frequency

Smart eyeglasses use cases. Smart eyeglasses are ideally suited for exploringand improving reading habits, because
many people already wear eyeglasses
for reading. In the most basic scenario,
a smart eyeglasses app could quantify
a wearer’s reading—for example, bycounting words. For simple reading
tracking using glasses, EOG is a viablechoice, because electrodes can be integrated into the eyeglass frame. Estimatedreading performance can be forwarded
to a smartphone and shared on a Web
portal. Users could compare their performance to that of their friends. We expect that tracking word count will havea similar effect as tracking step counts
using a pedometer—increasing the overall reading volume of users. For this purpose, we implemented a Web portal totest a quantified feedback approach.
A smart eyeglasses app could detect
document type and how documents are
read—determining, for example, concentration and attention levels related
to certain sections—providing insight
into healthy reading habits and helping
users revisit vital information that theymight have missed. Moreover, associating a comprehension level with text
sections could help authors improvetheir writing or could provide feedback
about readers’ progress.

5.0

0.7

4.5

0.6
0.5

4.0

0.4

3.5

0.3
0.2

1

2

Level

3

4

3.0

1

2

Level

3

4

Figure 3. Recognizing cognitive workload. Increasing task difficulty (from L1 to L4)
also increases changes in normalized blink frequency (measured by J!NS MEME
EOG) and decreases pupil diameter (measured by a stationary eye tracker). Average
results over eight repetitions per participant are shown. Correlations with cognitive
workload were observed.

tion, horizontal/vertical element of saccades). Details can be found elsewhere.19
We counted line breaks by recognizing
long saccades (eye movements) against
the dominant reading direction and approximated word count, either by assuming a fixed word count per line orby estimating word count from the average saccade number and length in a line
(see Figure 4b). Our analysis showed
an error rate of 6 to 11 percent whenreading 20 to 30 minutes from different
display sizes.19 There were 10 participants, including university students andstaff (four female and six male, between
ages 20 and 36), who were screened forpotential reading disabilities before thestudy. Although eyeglasses were notused here, we considered features that
could be derived from EOG-enabled

smart eyeglasses too. Therefore, we believe that the approach could be transferred to smart eyeglasses.
Furthermore, saccade features arewell suited for classifying whether a user
is reading a textbook (Figure 4c), comic(Figure 4d), fashion magazine, or newspaper.20 Our document type classification is based on eye movement featurestoo (saccade direction, slope, length,
and general reading direction, summarizing saccade direction over longer
times). We evaluated our approach in astudy with eight participants (three females and five males with a mean ageof 24) on the five document types listed
above. Recognition performance was
74 percent for u­ ser-independent training and 98 percent for user-dependent
training, as long as document layouts

PER VA SI V E computing9Digitally Enhanced Reality

saccade(a)fixation

(b)

(c)

(d)

ment between sleep-wake and day-nightcycles led to lower performance, memory,alertness, and gastrointestinal function.
In various studies, light exposure wasfound to be key in adjusting sleep-waketimes. Light in the morning increased
daytime alertness, and light during evening hours postponed sleep.22 Moreover,blue light has a stronger awakening effect than other wavelengths.
Current measurement practice. In freeliving circadian clock studies, light
exposure is currently measured using
wrist-worn sensor devices, necklaces,or specialized head-mounted light sensors. We found that wrist-worn sensors are often occluded by cloths, leading to substantial underestimation of
the actual light exposure. 23 Necklace
devices might similarly be hidden under garments, and thus incur measurement errors. Measuring light exposure
directly at the eyes is thus preferred byexperts.
Smart eyeglasses use cases. Smart eye-(e)Figure 4. Quantifying reading habits. (a) A participant wearing an SMI mobile eye
tracker while reading. (b) The camera view of the eye tracker with fixations andsaccades. The visual behavior recorded by the eye tracker for reading (c) a textbook
and (d) a comic. (e) A mock-up of a reading assistance service, providing quantified
feedback on reading habits (on a Web portal).

differed sufficiently (page size, relationship of words to images, two column versus one column layouts, and soon).20 Figure 4e shows a mock-up of areading assistance service, implemented
a Web portal.

10

PER VA SI V E computing 

Monitoring Light
Exposure Behavior
Light entering through the eyes cues ourcircadian phase and controls the humansleep-wake cycle.21 For example, in shift
workers, research found that a misalign-

glasses could measure light exposure in
daily life using a light sensor integrated
in rims or pad arm, hence close to theeyes but minimally occluded by hairor head. A smart eyeglasses app could
derive light-related behavior indicatorsand recommendations for efficient sleep
based on light measurement throughout a day. For example, to fall asleep
earlier at night, morning light exposure
should be increased. Consequently, theapp could recommend cycling to workor taking a longer walk outside during
an early lunch break. Moreover, blue
light exposure during late evenings, due
to TV or PC screen use, could be identified by the app.
Although smart eyeglasses serve hereas platform for sensors and processing,a smart eyeglasses app would forward
processed data to the Web portal, computer display, or smartphone for instantor regular user feedback. Consequently,
smart eyeglasses require communication functions (such as Wi-Fi or Bluetooth), but no visual user interface.

www.computer.org/pervasive

104

103

103
102
102
Blue irradiance

101

100
0

Green irradiance
Red irradiance101

Luminosity
200

400Time (s)Luminosity (lux)

RGB irradiance (mW/m2 nm)

Outdoors

(a)

600

800

100

104

103

103
102
102
101

100
0

Luminosity (lux)

RGB irradiance (mW/m2 nm)

Indoors

(b)

101

500

1,000
1,500
Time (s)

2,000

100

104

103

103
102
102
101

100
0

Luminosity (lux)

RGB irradiance (mW/m2 nm)

Screen work

(c)

101

200

400
Time (s)

600

100
800

Figure 5. Monitoring light exposure behavior via eyeglasses over a regular day. Activities and locations are illustrated in
irradiance/luminosity waveforms as colored areas: (a) Outdoors, luminosity is typically very high (≥ 5000 lux). (b) Indoors,
luminosity is much smaller than outdoors. (c) When viewing a screen work, luminosity and blue irradiance is elevated compared
to typical indoor conditions.

Initial analysis. We conducted a free-living study with 12 university students (five female and seven male, allbetween ages 18 and 28). Participants
continuously wore a custom light measurement device attached to regular

July–september 2015

eyeglasses for six consecutive days andannotated whether they were indoors
or outdoors. 23 Glasses were detachedonly when sleeping or showering. Participants who did not normally wear
eyeglasses were given dioptre-freee­yeglasses, individually adapted by
an optician. Light measurements weretaken continuously every 30 seconds.
Figure 5 shows wearing conditions
and example irradiance measurements
in RGB components from our study

PER VA SI V E computing 11

Digitally Enhanced Reality

the Authors
Oliver Amft is a full professor, heading the Chair of Sensor Technology andthe ACTLab research group at the University of Passau. His research focuses on
multimodal activity pattern recognition, human behavior inference algorithms,
and ubiquitous sensing systems, with applications in healthcare, sports, andbuilding automation. Amft received his PhD in electrical engineering and information technology from ETH Zurich. Contact him at amft@computer.org.

Florian Wahl is a PhD candidate in the Chair of Sensor Technology at the University of Passau. His research interests include ubiquitous sensing and patternrecognition. Currently, he is currently focused on sensing circadian phases.
Wahl received his MS in embedded systems from TU Eindhoven. Contact him
at wahl@ieee.org.

Shoya Ishimaru is a graduate student at Osaka Prefecture University. His research theme is activity recognition, which involves recognizing and classifying
human activities by using data from several sensors. Ishimaru received his BS
in engineering from Osaka Prefecture University. Contact him at ishimaru@m.
cs.osakafu-u.ac.jp

Kai Kunze is a project associate professor at the Graduate School of Media
Design, Keio University, Yokohama, Japan. His current research interest is cognitive activity recognition, with a focus on eyewear computing. Kunze received
his PhD in computer science from the University of Passau. Contact him at kai.
kunze@gmail.com

data. As the waveforms illustrate, outdoor light intensity can be 100 times
higher than indoors. TVs or computer
screens increase blue light components
and luminosity at the head to approximately 300 lux or more, thus muchhigher luminosity than otherwise foundafter sunset. A Wilcoxen ranksum test
across all study participants showed
highly significant differences (α = 0.01,
p = 0) between indoor and outdoor luminosity in daytime. We expect thatsmart eyeglasses will become a monitoring accessory to align the sleep-wakecycle in regular daily life. In particular,
users with varying work schedules, suchas airline flight staff, will benefit fromlight accounting and could use smart
eyeglasses to adjust their circadianphase ahead of time for their next destination’s time zone.

12

PER VA SI V E computing W

ith a catalogue ofsoftware apps to select from, integrated
multipurpose smart
eyeglasses could be used in different
applications and move from niche tomainstream markets. Yet several technical challenges must be further explored to realize the vision of smart
eyeglasses, including sensor design
and integration, low-power processing, overall ergonomic design, andd evising adequate interaction ap­
proaches. Unobtrusiveness will be
key to user adoption. Size, weight,
weight balance, and power consumption constraints will prevent designers
from integrating too many functional
components in one eyeglass device—especially if the glasses are intended
for continuous use daily. Considering

the identified hardware clusters, werarely expect existing interaction modalities to be integrated in smart eyeglasses. Instead, novel lightweight andlow-power interaction approaches for
smart eyeglasses need to be developed,
such as audio and vibration feedback.
Given suitable communication interfaces and interoperability, smart eyeglasses will also exploit use externalinteraction devices.
Furthermore, smart eyeglasses need
to overcome psychological and social
hurdles. The examples of J!NS MEME,
Optalert, and EnWake smart eyeglasses
suggest that wearers accept continuous
sensing when they identify clear benefits from using the device. Nevertheless,qualitative studies are needed to investigate wearer opinions when using smart
eyeglasses. Social hurdles include both
the wearer’s fear of being stigmatizedand privacy concerns of those aroundthem. In part, social hurdles might beovercome if more smart eyeglasses become available, minimizing privacy
concerns by not including cameras.
It is interesting to note that when
we informally asked our nontech colleagues about the idea of continuously
worn smart eyeglasses, those who do
not normally wear eyeglasses reacted
negatively. However, after explaining
the potential uses and assistance functions, people often agreed that the benefits were convincing. We neverthelessbelieve that initial integrated multipurpose smart eyeglasses will best serve select application fields, including healthand interaction—particularly as related
to sports—and cognitive aid.

Acknowledgements
This research was partly supported by the Dutch
Technology Foundation STW under grant numbers 12184, Project OnTime.

References
1. B.E Stein, T.R. Stanford, and B.A. Rowland, “The Neural Basis of Multisensory
Integration in the Midbrain: Its Organization and Maturation,” Hearing Research,
vol. 258, no. 1, 2009, pp. 4–15.

www.computer.org/pervasive2. O. Amft and G. Tröster, “On-Body Sensing Solutions for Automatic Dietary Monitoring,” IEEE Pervasive Computing, vol.
8, no. 2, 2009, pp. 62–70.
3. G.J. Siegle, N. Ichikawa, and S. Steinhauer, “Blink Before and After You
Think: Blinks Occur Prior to and Following Cognitive Load Indexed by Pupillary
Responses,” Psychophysiology, vol. 45,
no. 5, 2008, pp. 679–687.

Symp. Wearable Computers (ISWC),
2008, pp. 286 –291; doi: 10.1109/
ISWC.2008.4911575.
10. M. Kourogi and T. Kurata, “Personal
Positioning Based on Walking Locomotion Analysis with Self-Contained Sensors and a Wearable Camera,” Proc. 2ndIEEE/ACM Int’l Symp. Mixed and Augmented Reality, 2003, p. 103.

4. L. Atallah et al., “Gait Asymmetry Detection in Older Adults Using a Light EarWorn Sensor,” Physiological Measurement, vol. 35, no. 5, 2014, pp. N29–N40.

11. R.W. DeVaul et al., “The Memory Glasses:
Subliminal vs. Overt Memory Support
with Imperfect Information,” Proc. 2012
16th Int’l Symp. Wearable Computers,
2003, pp. 146–146.

5. C. McLaughlin, K. Moffitt, and J. Pfeiffer,“P-21: Human Factors Guidelines for Binocular Near-Eye Displays,” SID Symp.
Digest of Technical Papers, vol. 34, no. 1,
2003, pp. 280–283.

12. A. Bulling, J.A. Ward, and H. Gellersen,“Multimodal Recognition of Reading
Activity in Transit Using Body-Worn Sensors,” ACM Trans. Applied Perception,
vol. 9, no. 1, 2012, pp. 2:1–2:21.

6. L. Bouarfa et al., “Predicting Free-Living
Energy Expenditure Using a Miniaturized
Ear-Worn Sensor: An Evaluation Against
Doubly Labelled Water,” IEEE Trans.
Biomedial Eng., vol. 61, no. 2, 2014, pp.
566–575.

13. H. Manabe and M. Fukumoto, “FullTime Wearable Headphone-Type Gaze
Detector,” Extended Abstracts on
Human Factors in Computing Systems(CHI), 2006, pp. 1073–1078.

7. J.A.C. Patterson, D.G. McIlwraith, andG.-Z. Yang, “A Flexible, Low Noise
Reflective PPG Sensor Platform for EarWorn Heart Rate Monitoring,” Proc. 6thInt’l Workshop Wearable and Implantable Body Sensor Networks (BSN), 2009,
pp. 286–291; doi: 10.1109/BSN.2009.16.
8. E. Costanza et al., “Eye-q: Eyeglass
Peripheral Display for Subtle Intimate
Notifications,” Proc. 8th Conf. HumanComputer Interaction with Mobile
Devices and Services, 2006, pp. 211–218;
doi: 10.1145/1152215.1152261.
9. B. Cinaz and H. Kenn, “HeadslamSimultaneous Localization and Mappingwith Head-Mounted Inertial and Laser
Range Sensors,” Proc. 12th IEEE Int’l

July–september 2015

14. O. Amft et al., “Analysis of Chewing
Sounds for Dietary Monitoring,” Proc. 7th
Int’l Conf. Ubiquitous Computing, 2005,
pp. 56–72; doi: 10.1007/11551201_4.
15. Y. Ishiguro et al., “Aided Eyes: Eye Activity Sensing for Daily Life,” Proc. 1st Augmented Human Int’l Conf., 2010, article
no. 25; doi: 10.1145/1785455.1785480.

18. D.W. Moore, D.E Alvermann, and K.A.
Hinchman, Struggling Adolescent Readers: A Collection of Teaching Strategies,
Int’l Reading Assoc., 2000.
19. K. Kunze et al., “The Wordmeter—Estimating the Number of Words Read Using
Document Image Retrieval and Mobile
Eye Tracking,” Proc. 12th Int’l Conf.
Document Analysis and Recognition(ICDAR), 2013, pp. 25–29; doi: 10.1109/
ICDAR.2013.14.
20. K. Kunze et al., “I Know What You Are
Reading: Recognition of Document
Types Using Mobile Eye Tracking,”
Proc. 2013 Int’l Symp. Wearable Computers (ISWC), 2013, pp. 113–116; doi:
0.1145/2493988.2494354.
21. J.F. Duffy and K.P. Wright, Jr., “Entrainment of the Human Circadian System byLight,” J. Biological Rhythms, vol. 20, no.
4 2005, pp. 326–338.
22. V.L. Revell and C.I. Eastman, “How to
Trick Mother Nature into Letting You
Fly Around or Stay Up all Night,” J. Biological Rhythms, vol. 20, no. 4, 2005, pp.
353–365.
23. F. Wahl, T. Kantermann, and O. Amft,“How Much Light Do You Get? Estimating Daily Light Exposure Using Smartphones,” Proc. 2014 ACM Int’l Symp.
Wearable Computers (ISWC), 2014, pp.
43–46; doi: 10.1145/2634317.2634346.

16. C.-T. Lin et al., “Wireless and Wearable
EEG System for Evaluating Driver Vigilance,” IEEE Trans. Biomedical Circuits
and Systems, vol. 8, no. 2, 2014, pp. 165–
176.
17. A.E. Cunningham and K.E. Stanovich,“What Reading Does for the Mind,” J.
Direct Instruction, vol. 1, no. 2, 2001,
pp. 137–149.

Selected CS articles and columns
are also available for free at
http://ComputingNow.computer.org.

PER VA SI V E computing 13