IN V ISIBL E C OMP U T ING

Activity Recognition
for the Mind:
Toward a Cognitive“Quantified Self”
Kai Kunze, Masakazu Iwamura, andKoichi Kise, Osaka Prefecture University
Seiichi Uchida, Kyushu University
Shinichiro Omachi, Tohoku University

Applying mobile sensing technology to cognitive tasks will enable novel
forms of activity recognition.

Physic a l act ivit y recognit ion
technology has become
mainstream—many dedicatedmobile devices and smartphone
apps count the steps we climb or
the miles we run. What if devices
and apps were also available that
could count the words we readand how far we’ve progressed in
our learning? The authors of thisarticle demonstrate that mobile
eye tracking can be used to do justthat. Focusing on reading habits,
they’ve protot yped cognitive
activity recognition systems thatmonitor what a nd how muchusers read as well how much they
understand. Such systems could
revolutionize teaching, learning,
and assessment both inside andoutside the classroom. Further,as sensing technology improves,activit y recognition could beextended to other cognitive tasks
including concentrating, retaining
information, and auditory or visualprocessing. While this research isextremely exciting, it also raises
numerous ethical questions—forexample, who should know what weread or how much we understand?
Albrecht Schmidt, column editor

0018-9162/13/$31.00 © 2013 IEEEP

eople increasingly use
mobile computing
technology to track
their health and fitness
progress, from simple step counting
to monitoring food intake tomeasuring how long and well they
sleep. Smartphone applications
such as RunKeeper (http://runkeeper.com) and Lose It! (www.
loseit.com) and wearable devices
such as the Fitbit FLEX wristband(www.fitbit.com) foster better eatingand exercise habits, decrease therisk of obesity-related diseases, andimprove quality of life.
Activity-tracking abilities are still
hampered by the limited battery
power of today’s mobile devices, but
emerging technologies such as theM7 motion-sensing coprocessor in
the new iPhone 5s make it easierto aggregate and interpret sensor
data in a power-efficient manner.
In addition, while most activityrecognition apps continue to focus
on physical movement, such assteps taken or stairs climbed, newproducts such as Withings Pulse(www.withings.com/en/pulse) also
quantify physiological signals suchas heart rate.
Given these trends, it’s only
a matter of time before we see
mobile sensing technology applied
to cognitive tasks, enabling novel
forms of activity recognition.

Published by the IEEE Computer Society

OCTOBER 2013

EYE MOVEMENTS AND
COGNITIVE TASKS
Sensing brain activity seems
to provide the most insight into
cognitive processes. However,doing so accurately requires
relatively invasive methods suchas electroencephalography (EEG),
functional magnetic resonanceimaging, and electrocorticography.
Of these, EEG is the most promising
for mobile applications, as evidenced
by forthcoming products such as theEmotiv Insight neuroheadset (http://emotivinsight.com). Yet, EEG is quite
noisy and easily overshadowed by85

IN V ISIBL E C OMP U T ING

Figure 1. Mockup interface for a hypothetical Readit service that records and analyzes
reading habits, much like a fitness service such as Fitbit.

muscle movement, requiring timeconsuming and difficult signalprocessing.
An alternative approach is to track
eye movements, which are strongly
correlated with cognitive tasks.
However, because eye movements
include information about the user’sattentiveness, degree of fatigue,
emotional state, and so on, it canbe difficult to isolate the objectof interest. Nevertheless, early
research showed that stationary eye
trackers can differentiate various
types of media consumption—forexample, watching videos, surfing
the Web, and reading magazines
have different eye-movement
characteristics (H. Drewes andA. Schmidt, “Interacting with theComputer Using Gaze Gestures,”Proc. 11th IFIP TC Int’l Conf. HumanComputer Interaction [Interact 07],
LNCS 4463, Springer, 2007, pp.
475-488).

MOBILE EYE TRACKINGElectrooculography (EOG)
and optical eye tracking are two

86

COMPUTER

promising methods to record eye
gaze in real-life mobile settings.
The eye can be represented asa dipole between the cornea andretina, and EOG uses electrodes
to measure the change in potential when the eye moves. With thistechnique, it’s possible to recognize
reading and other eye movements
in realistic everyday scenarios. Although the electrodes to measure
EOG can be easily embedded inglass frames to make them more
unobtrusive, they still need totouch the skin near the eye, which
can be uncomfortable. On theother hand, this approach is cheap
to implement and requires littleprocessing.
An alternative option is to track
eye gaze using infrared light andstereo cameras, which provides
potentially higher accuracy but requires more processing power. This
type of system infers eye motion
and gaze pointing based on irisshape, sometimes in combination
with local face features such as eyelids and eye corners.

Commercial mobile eye trackers
are still quite expensive. Yet, giventhat the hardware—electrodes,cameras, and infrared LEDs—
is relatively inexpensive, prices
should fall as demand increases
just as they have for head-mounted
displays since Google Glass wasannounced. An example of a cheap
3D-printed eye tracker was shownat this year’s MobileHCI conference(K. Lukander et al., “OMG!: A New
Robust, Wearable and Affordable
Open Source Mobile Gaze Tracker,”
Proc. 15th Int’l Conf. HumanComputer Interaction with Mobile
Devices and Services, ACM, 2013,
pp. 408-411).
Patent applications suggestthat Google, Apple, and other techcompanies are already trying out
mobile eye-tracking prototypes
internally. Researchers are also
experimenting with front cameras
on tablets and smartphones tocarry out low cost, low-fidelity eyegaze detection (K. Kunze et al., “MyReading Life: Towards Utilizing
Eyetracking on Unmodified Tabletsand Phones,” Proc. UbiComp ’13
Adjunct, ACM, 2013, pp. 283-286).

QUANTIFYING READINGHABITS
Measuring reading activity is one
way to quantify mental “fitness.”
Although an increase in reading
volume leads to demonstrably improved language comprehension
and critical thinking skills, few eyetracking researchers have evaluatedreading habits in situ (A. Bulling,J.A. Ward, and H. Gellersen, “Multimodal Recognition of Reading
Activity in Transit Using Body-Worn
Sensors,” ACM Trans. Applied Perception, vol. 9, no. 1, 2012, article no. 2).
In our research, we’re exploring
the possibility of using mobile eyetracking technology to determine
how much people read, what type ofdocument they’re reading, and how
much of what they’re reading thatthey understand.Just as fitness apps record andanalyze physical activity to help
users get into better shape, a“reading habits” app could recordand analyze the number of words
read, types of documents read, andperiods of concentrated reading tohelp users become better thinkers.
Figure 1 shows an interface
mockup for a hypothetical Readit
service.

Counting read words
Toward this end, we’ve createdthe Wordometer app for the SMI
iView X eye tracker, shown in
Figure 2, that estimates the numberof words a person reads, much
like a pedometer counts steps(K. Kunze et al., “The Wordometer—
Estimating the Number of Words
Read Using Document Image
Retrieval and Mobile Eye Tracking,”
Proc. 12th Int’l Conf. Document
Analysis and Recognition [ICDAR13], IEEE CS, 2013, http://kaikunze.
de/papers/2013Kunze-1.pdf).
To develop the app, we firstrecorded the eye movements ofnumerous users reading English
documents and then applied aline-break detector to the data thatlooked for a long saccade (fast eyemovement) against the main left-toright reading direction. Finally, we
created a support vector regressionalgorithm that inputs detected line
breaks and some eye-gaze features
to estimate how many words a user
has read. Our word-count estimation
method achieves an average error
rate of only 6.5 percent for 8 users
over 10 documents.

Distinguishing document
types
Counting read words is just one
aspect of understanding reading
habits. We’re also using mobile eye
tracking to distinguish different
document types (K. Kunze et al.,“I Know What You Are Reading—
Recognition of Document Types
Using Mobile Eye Tracking,” Proc.

Eye tracker

Figure 2. Using mobile eye tracking to count read words. Left: A user reads while
wearing the SMI iView X eye tracker. Lower center: The eye tracker uses infrared lightand stereo camera views to track eye gaze. Right: The Wordometer application maps
the user’s eye gaze to the document he’s reading. The lines show saccades, and thecircles indicate where users fixated their gaze. The size of the circles correlates withthe duration of the fixations.

17th Ann. Int’l Symp. Wearable
Computers [ISWC 13], ACM, 2013,
pp. 113-116; http://kaikunze.de/papers/2013Kunze-5.pdf).
In particular, we analyze saccade
directions to obtain information
about how much structured text
versus images are on a page. The
slope of the eye gaze indicates
general reading direction, anda percentile distance gives anapproximation of page size.
To assess this technique’s
effectiveness, we conducted astudy in which eight users wearing
the iView X eye tracker in fivedifferent environments read fivedifferent types of Japanese readingmaterial (novel, manga, magazine,newspaper, and textbook),
spending about 10 minutes on eachdocument. This approach achieved
a 99 percent user-dependent and74 percent user-independent
recognition rate over a window
size of about 1 minute of gazedata. This research represents a
crucial first step toward the visionof wearable reading assistants andreading logs.

Assessing reading
comprehension
In addition, we’re exploring
whether eye tracking canbe used to measure reading

comprehension—specifically,second-language skills (K. Kunze
et al., “Towards Inferring Language
Expertise Using Eye Tracking,” CHI’13 Extended Abstracts on Human
Factors in Computing Systems [CHIEA 13], ACM, 2013, pp. 217-222).
We had 10 native Japanese
speakers learning English asa second language read 10
textual passages and answer theaccompanying comprehension
questions from the Test of Englishfor International Communication(TOEIC) while wearing the iView Xeye tracker. The participants sat at aregular desk with the test in front ofthem and had about 20 minutes tocomplete each task, after which we
asked them to mark those words inthe passage they didn’t understand.
We recorded the subjects’ eyegaze and mapped this data todigitized versions of the textual
passages using an image-retrieval
method based on locally likely arrangement hashing (LLAH), as
Figure 3 shows. Figure 4a depicts a
raw eye trace recorded by the eyetracker laid over the digital document using LLAH and manual line
segmentation. Horizontally projecting the read line results in thefixation points shown in Figure 4b.
This makes it possible to calculate ahistogram of the eye fixations thatOCTOBER 201387

IN V ISIBL E C OMP U T ING

Image from eye tracker

Retrieved pageDocument ID
Read word

Recognized event
Figure 3. Mapping reading activity recorded using the eye tracker to a digitizedversion of the physical document retrieved from a database.

Kai Kunze is an assistant professor in the Department of Computer
Science and Intelligent Systems at
Osaka Prefecture University, Japan.
Contact him at kai.kunze@gmail.
com.

(a)

Masakazu Iwamura is an associate
professor in the Department of Computer Science and Intelligent Systems
at Osaka Prefecture University. Contact him at masa@cs.osakafu-u.ac.jp.

(b)

(c)
Figure 4. Visualization of eye-movement on a single sentence based on therecording with the eye tracker. (a) Raw trace of eye movements laid over the digital
document; the red dots indicate fixation points and the blue lines indicate saccades.
(b) Processed eye-movement data: the fixations are filtered and aligned to the text
line to provide insights into reading speed and time spend on words. (c) Histogram
showing aggregated time spent reading parts of the text; in this case, the user didn’t
understand the work “mortgages” well.

reveals what words the reader had
trouble with, as Figure 4c shows. In
this case, the reader had special difficulty understanding “mortgages.”
The method worked for all words
marked as difficult by the participants. Future research will study
how to use eye gaze to infer readers’
TOEIC score.

88

COMPUTER

gies to optimize mental fitness andwell-being.
If educators could obtain feedback about their students’ attention
level in class and learning progress,for example, they could modify theirpresentation and tailor learning material to individual pupils’ needs,
backgrounds, and preferences. Content creators could likewise leveragequantified “mind logs” to improvetheir works: Which part of a movie
most excites users? What feelings
does a particular paragraph in abook convey? At what point in a
grant proposal does the reader loseinterest? And, given enough users
and high-quality sensor data, researchers could even address more
complex lifestyle issues, such as
how sleep and eating habits influence alertness and learning.

Although our research
has focused on reading,the methodology we’vedeveloped can be applied to a wide
variety of cognitive tasks. Our overall goal is to facilitate long-term
tracking of all types of mental processes, which will enable new forms
of self-reflection and suggest strate-Koichi Kise is a professor in the Department of Computer Science andIntelligent Systems at Osaka Prefecture University. Contact him at
kise@cs.osakafu-u.ac.jp.
Seiichi Uchida is a professor in
the Department of Advanced Information Technology at Kyushu
University, Japan. Contact him atuchida@ait.kyushu-u.ac.jp.
Shinichiro Omachi is a professor in
the Department of Communications
Engineering at Tohoku University,
Japan. Contact him at machi@ecei.
tohoku.ac.jp.

Editor: Albrecht Schmidt, University ofStuttgart, Germany; Albrecht@computer.org