Position Paper: Brain Teasers –Toward Wearable Computing thatEngages Our Mind
Shoya Ishimaru

Abstract

Graduate School of Engineering
Osaka Prefecture University
Sakai, Osaka, Japanishimaru@m.cs.osakafu-u.ac.jp

The emerging field of cognitive activity recognition – reallife tracking of mental states – can give us new
possibilities to enhance our minds.

Kai Kunze

In this paper, we outline the use of wearable computing toengage the user’s mind. We argue that the more personaltechnology becomes the more it should also adopt to theuser’s long term goals improving mental fitness. We
present a the concept of computing to engage our minds,
discuss some enabling technologies as well as challenges
and opportunities.

Graduate School of Engineering
Osaka Prefecture University
Sakai, Osaka, Japanfirstname.lastname@gmail.com

Koichi Kise
Graduate School of Engineering
Osaka Prefecture University
Sakai, Osaka, Japan
kise@cs.osakafu-u.ac.jp

Masahiko Inami
Graduate School of Media Design
Keio University
Yokohama, 223-8526 Japaninami@kmd.keio.ac.jp
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. Copyrights for componentsof this work owned by others than ACM must be honored. Abstracting withcredit is permitted. To copy otherwise, or republish, to post on servers or toredistribute to lists, requires prior specific permission and/or a fee. Request
permissions from permissions@acm.org.
UbiComp’14 Adjunct, September 13-17, 2014, Seattle, WA, USA
Copyright is held by the owner/author(s). Publication rights licensed to ACM.
ACM 978-1-4503-3047-3/14/09. . . $15.00.
http://dx.doi.org/10.1145/2638728.2641722

Author Keywords
Intelligence Amplification, Wearable Computing, Mind,
Cognitive Activity Recognition

ACM Classification Keywords
H.5.m [Information interfaces and presentation (e.g.,HCI)]: Miscellaneous.

Introduction
So far we are mostly augmenting our minds, externalizing
it on documents, schedules, to-do lists, calendars etc. The
speed this happens increased with the advent of thepersonal, pervasive and wearable computing.
Offloading appointments, birthdays, thoughts etc. gives

us the ability to free our mind and to focus on other more
abstract ideas. Yet, it also created an attention economy,in which it can be increasingly hard to focus andremember things without our analog or electronic
tools.We get more and more dependent on technology.
There are even some researchers arguing that we will lose
the ability to remember and critical think [6].
Leveraging the advances in cognitive science, psychology
and neuroscience we should be able to track attention,
focus, engagement and other brain activities in realisticenvironments. We can use these cognitive logs to improve
our cognitive skills while performing a task. In this paper,
we discuss how it can be possible to design interactions
with computing is such a way that it supports, yet also
gives us a certain autonomy and challenge: Computing toengage our Minds.
We believe that tracking cognitive activities and assessing
cognitive load, attention, fatigue etc. can help us build
computing systems that on the one hand support us
during everyday life on the other hand enhance ourintellect.
In the following we describe the concept of ”wearable
computing that engages our minds”, go into some details
about the enabling technologies and finally discuss somechallenges we see.

Concept
Instead of just focusing on ease of use and pro-active
computing (e.g. inferring what the user wants without
direct interaction), we could also incorporate concepts andtools for cognitive self-improvement. There are already
ample of applications that try to improve your mental
skills, the process is usually named ”brain jogging”.

However, as studies show these games have limited effectto real world brain performance.
We propose that the user improves her cognitive skills
actively while being supported by a pervasive computingsystem. The system in turn regulates the difficulty andtask granularity based on the mental state of the user andher goals.
For example take Naomi, she wants to improve her spatial
and navigation skills for a trip to the Amazon. She’s
currently in Zurich and wants to visit a friend using anavigation application on her wearable system. As she still
has around 2 hours time before the meeting, the wearable
navigation system could make her trip harder by giving
her just visual landmarks to navigate by in turn training
her navigation skills.
As another example take Jim, who has first stages of age
related dementia. He has a hard time remembering what
he did over last weeks. He’s wearing a life logging systemand his pervasive home system picks the most memorable
pictures from the last weeks and displays them on thedigital picture frames in the house at a time when hismental state suggests he’s active and able to process theinformation.
As we understand learning better in real life environments,
this can lead to a new way of teaching and change
education as we know it. We believe that the concept of”Engaged Computing” is not only beneficial in educationalsetting but also for a lot of other tasks we perform during
our life. Cognitive activity recognition promises to detect
the current mental capacity of the user. Imagine, keeping
a car driver or pilot healthily engaged with their vehicle sothey can control it better in case of an emergency.

Enabling Technologies
Cognitive Activity Recognition
If we can quantify attention, focus, engagement,concentration and similar mental states, we can track
cognitive progress/decline and build systems that canimprove our mental abilities. Therefore, cognitive activity
recognition is crucial [3, 4].
For example, near-infrared spectroscopy (NIRS) can
measure the blood hemoglobin concentrations and giveinformation about brain activity. We use the device to get
the ground truth for the cognitive load of the user coupledwith other sensing modalities (e.g. head motion, eyetracking).

Figure 1: Experimental Setup ofthe LabNIRS device fromShimadzu and the brain activityof one participant for the n-backtask with increasing difficulty. As
seen from the heat maps, the last
most difficult task shows lessbrain activation.

In Figure 1, we see the experimental setup, a user wearing
a near-infrared spectroscopy (NIRS) device (42 channels).
We depict also the brain activity in the prefrontal cortex
of a participant doing a memory game (dual n-back) withan increasing difficulty. As seen from Figure 1, the brain
activity increases steadily from easy to hard. Yet, for thelast, hardest case we have a drop in brain activity (dualn-back, n=4). The user is overexerted in this case andcannot cope with the problem anymore, leading to a
reduced oxygen level in the brain.
Therefore, a system trying to improve the user’s memory
should adjust the difficulty to n=3 to engage the user’s
mind optimally.
As brain imaging techniques, eye trackers and othercognitive task tracking devices are still expensive. We
need to focus on alternative technologies that are alreadycommercially available or can be easily integrated into
consumer devices, to enable cognitive task tracking. One
way to approach this problem, is to provide affordable
commodity devices. Head-mounted display computers,

most prominently Google Glass, seem a perfect tool for
cognitive task analysis, as they are already worn on thehead (close to the brain) enabling easy sensor integration.
A very simple sensor (infrared distance sensor from Google
Glass) can for example measure eye blinks. Astonishingly,
blinking frequency alone is already able to distinguish a
couple of cognitive tasks (e.g. Reading versus Talking to aperson [1]). And we see also new devices that focus on
cognitive activity recognition, for example J!NS MEME
(see Figure 2, smart glasses that use Electrooculography
to detect eye movements recognizing fatigue).
Life Logging as Memory Enhancement
Most life-logging technologies focus on capture and accessscenarios to index and retrieve information at a later point
in time. However, We can also use this technology incombination with cognitive activity recognition to boost
our brain (see Jim’s Scenario).
There is already research by Isola et al. analyzing whatmakes images memorable [2]. If we build on this work andcombine it with life-logging and measures for user’s
attentiveness and the cognitive load while looking at thepicture, we can build systems that make it easier toremember situations, people and places. Here again
cognitive load estimation is important, especially while theuser sees the picture. With this feedback the system candecide when to show the picture again.

Discussion and Challenges
Miniaturization – as seen from one of our currentexperimental setups (Fig. 1), a lot of the brain imaging
devices are still too large and cumbersome to be worn
during everyday life. Yet, current research efforts alreadyfocus on making them useable during everyday life [5].
Exploration of Sensing Modalities – Initial research showsthat especially eye tracking but also other pervasive
sensing modalities can give insights into cognitive
activities. Using brain imaging techniques for groundtruth we should explore other sensors.

How can we introduce the concept of computing thatengages our mind without risking the safety of people andwithout frustrating the user?

Acknowledgements
Cognitive Computing in the large – As our brains are very
versatile, there is large variability between users.
Therefore, we need large scale, well-designed user studies
to better understand the relations between cognitive load,
task and applicable sensing. As with wearable computing
in general, large representative datasets are required.

Figure 2: User wearing smart
glasses able to track fatigue andother mental states related to eyemovement.

Stress Versus Engagement – This challenge is again more
related to the work-load measurements: Can we
distinguish between healthy engagement of and stressful,unhealthy straining of our mind? Challenging the user too
often creates the danger of stress and might lead
eventually to burn-out. We will need a new level ofquality for the detection algorithms and maybe a novel
decision making process for systems tracking cognitive
tasks, as simply an accuracy of 90% (ok for most physicalactivity recognition tasks) is not be good enough toprevent these problems.
Mind Improvement?– As we still struggle with cognitive
sensing, quantifying cognitive tasks in everyday life, it’s
hard to tackle the question about effective interventions
to improve our mind. We should begin analyzing theimpact cognitive activity tracking of usability, human
computer interaction and design.
Interaction Design – For learning tasks and memory
games, the interaction design for ”engaged computing”
seems straight forward: As long as the user seems
mentally fit and brain activity increases, make the task
more difficult. Yet, this might be dangerous for a lot ofreal world activities (maintenance, transportation etc.).

We would like to thank to J!NS and Shimadzu forsupplying us with LabNIRS and J!NS MEME prototypes.
This is work is partly supported by the CREST project.

References[1] Ishimaru, S., Kunze, K., Kise, K., Weppner, J.,
Dengel, A., Lukowicz, P., and Bulling, A. In the blinkof an eye: combining head motion and eye blink
frequency for activity recognition with google glass. In
Proc. of AH (2014).
[2] Isola, P., Xiao, J., Torralba, A., and Oliva, A. Whatmakes an image memorable? In Proc. of CVPR
(2011).
[3] Kunze, K., Iwamura, M., Kise, K., Uchida, S., andOmachi, S. Activity recognition for the mind: Towarda cognitive “quantified self”. Computer, 10 (2013).
[4] Kunze, K., Kawaichi, H., Yoshimura, K., and Kise, K.
The wordmeter – estimating the number of words readusing document image retrieval and mobile eye
tracking. In Proc. ICDAR 2013 (2013).
[5] Peck, E. M., Afergan, D., and Jacob, R. J.
Investigation of fnirs brain sensing as input toinformation filtering systems. In Proc. of AH, ACM(2013).
[6] Subrahmanyam, K., Michikyan, M., Clemmons, C.,
Carrillo, R., Uhls, Y. T., and Greenfield, P. M.
Learning from paper, learning from screens: Impact ofscreen reading and multitasking conditions. Int.
Journal of Cyber Behavior (IJCBPL) (2013).