Late-Breaking Work

CHI 2017, May 6–11, 2017, Denver, CO, USA

Facial Thermography for Attention
Tracking on Smart Eyewear: An Initial
Study
Benjamin Tag
George Chernyshov
Keio University Graduate School Keio University Graduate Schoolof Media Designof Media Design
Yokohama, Japan
Yokohama, Japantagbenja@kmd.keio.ac.jp
chernyshov@kmd.keio.ac.jp
Ryan Mannschreck
Naohisa Ohta
Keio University Graduate School Keio University Graduate Schoolof Media Designof Media Design
Yokohama, Japan
Yokohama, Japanrmann@kmd.keio.ac.jp
naohisa@kmd.keio.ac.jp
Kazunori Sugiura
Kai Kunze
Keio University Graduate School Keio University Graduate Schoolof Media Designof Media Design
Yokohama, Japan
Yokohama, Japanuhyo@kmd.keio.ac.jp
kai@kmd.keio.ac.jp

Abstract
We are describing the first step towards the developmentof an unobtrusive open eyewear system for attention tracking in daily life situations. We are logging thermographic
data from infrared imaging and electrooculographic readings from off-the-shelf smart glasses and measure cognitive engagement of people in different situations. We are
identifying new potential areas on the face for contactless
IR temperature sensing. Attached to smart glasses and incombination with the EOG potentials we can monitor thewearer’s facial temperature changes, eye movement andeye blink in everyday situations, which is a major step towards becoming able to measure attention in unconstrained
settings, and thus make it manageable.

Author Keywords
Thermography; Electrooculography; Attention; Psychophysiology; Eyewear; Tracking; Sensing

ACM Classification Keywords
H.5.2 [User Interfaces ]: User-centered design
Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the Owner/Author. Copyright is held by theowner/author(s). CHI’17 Extended Abstracts, May 06-11, 2017, Denver, CO, USA
ACM 978-1-4503-4656-6/17/05. http://dx.doi.org/10.1145/3027063.3053243

Introduction and Motivation
Users of smart devices, phones, tablets and a growing
number of Internet of Things appliances have to deal withan increasing amount of data presented to them, such as
floods of notifications. This has resulted in an explosive2959

Late-Breaking Work

CHI 2017, May 6–11, 2017, Denver, CO, USA

growth of information available at virtually any time and any
place, which leads to a situation where we are exposed tomore information than we can process. But where lies thebenefit in a constant exposure to available information anddata, if you forget the one crucial piece of information suchas a deadline? It becomes clear that we have to learn tomanage and economize our use of attention every day, andallow to replenish our attention reservoir. There are several
ways of subdividing attention into more defined categories,
e.g. volitional attention and non-volitional attention. For our
research purposes we decided to follow the The Merriam
Webster dictionary which defines attention as "the act orstate of applying the mind to something" and as "a condition of readiness for such attention involving especially aselective narrowing or focusing of consciousness and receptivity" [9].

Figure 1: J!NS MEME and closeup
on the EOG electrodes

In this paper we discuss the first step towards a wearable,
everyday device that tracks attention. We are using physiological expressions of cognitive activity to identify perceptual processes, such as attention. This will allow us tomeasure attention and therefore will make the developmentof effective tools and applications for attention management possible. We are utilizing facial thermography measurements from infrared imaging and Electrooculography(EOG) data obtained via smart eyewear (J!NS MEME, Figure 1 [6]). So far, facial temperature has been measured
in locations on the head that would render an unobtrusive,
everyday wearable sensing device almost impossible tobe realized. Therefore, we aim at identifying locations fortemperature measurements that are adjacent to the area
covered by standard off-the-shelf spectacle frames. Consequently, the development of attention tracking smart wear
would enable future work in real life situations, and liberate
subjects and researchers from expensive medical grade
equipment.

Related Work
One approach to estimate cognitive workload, that does
not directly come to mind, is based on measuring the temperature differences between certain facial areas [4, 8].
Studies investigating this topic have already been done[17,7, 5], unexceptionally concentrating on stationary setups
rather than a wearable device design though. It is proventhat in times of higher cognitive load (e.g. studying) cerebral neurons are more active than in resting states. Thismeans, they require more oxygen and glucose to function.
Around 1/3 of the energy produced by oxygen and glucose
based chemical reactions is released as heat, i.e. the brain
produces more heat when it is in higher demand. Fat andbones of the skull act as heat insulators, which significantly
decreases the importance of convective cooling, making theblood circulation the main heat exchange mechanism[15,13].
In order to assess the changes of the brain temperature, we
compare temperatures of facial tissues supplied by bloodfrom the internal carotid artery, that is passing through thebrain with temperatures of tissues supplied by the external carotid artery. The common carotid arteries are themain blood supply of the head and branch into exterior andinterior carotid arteries in the throat region. The internal
carotid artery is supplying the brain and is connected to all
the main intracranial arteries via the Circle of Willis. Thus,
blood from the Circle of Willis can be used as an indicatorof the brain temperature. One of the branches of the internal carotid artery is the ophthalmic artery. It supplies theeyes and surrounding tissues, and has branches that supply nasal, eyebrow and forehead regions of the face. For
reference we use the other branch of the common carotid
artery, the external carotid artery, that is supplying the face
and the scalp. This vascular structure lets us compare thetemperature differences between tissues supplied by blood2960

Late-Breaking Work

CHI 2017, May 6–11, 2017, Denver, CO, USAfrom the internal carotid artery (the eye, parts of the nose,and lower part of the forehead) with the tissues supplied by
the external carotid artery (cheeks, temples, nasal area).
Thus we can estimate the temperature differences between
two branches of the common carotid artery: one that went
through the brain and another that did not. The change ofthis difference can give a significant clue about the cognitive
load.
J!NS MEME smart glasses are equipped with sensors thatallow the capturing of electrooculography (EOG) and enable
the accurate measurement of eye movements, and therefore detecting eye blinks [2]. Siegle et al. use blink rates asa measure for concentration, which is directly connected toattention [12]. In times of danger, the normal human reflex
is to suppress the natural eye blink in order to avoid missing
vital visual information. Similarly, special ocular tasks makepeople time their eye blink so that the amount of missed information can be kept as small as possible [10, 14]. Thismeans our attentional system has a direct influence on ourblink reflex.

Approach
Humans receive most of their sensory information throughtheir head, making it a particularly interesting location for
sensing, tracking, and enhancing social and cognitive functions. There are first indications from controlled lab studies that specific behavior patterns and physiological signals (e.g. eye movements, eye blinks, nose temperature,and head motions) are linked to those social and cognitive
functions [1]. However, most studies focus on constrainedlab settings with artificial simplified tasks using expensive
medical equipment. The use of unobtrusive head based
sensing to estimate cognitive functions in real life situations
is largely unexplored. Therefore, we believe that the form
factor of wearable electronics has a very high potential for

research in areas such as attention management and behavior change. Moreover, according to the National Eye
Institute, 64% of the adult population of the US are wearing eyeglasses [11]. This is significantly more than the 41%of Americans wearing watches [16]. This shows the great
potential of smart eyewear.
As introductorily described in the related works section,
there are patterns in human physiological signals (facial
temperature, eye blink, etc.) that can reveal information
about cognitive activities. Our main focus is to explore how
we can utilize this data in order to become able to better understand our behavior and make the allocation of attention
throughout the day manageable. Moreover, we want to enable people to optimize their limited cognitive resources fortheir own personal and/or professional goals.
We based this study on the related research and are gathering information on relative temperature changes of different areas of the face during small tasks that require acertain cognitive activity. We will later use this data to selectthe most appropriate locations for the placement of contactless temperature sensors on our wearable hardware for
future works.

Experiment Design
To investigate the correlation of changes in facial temperature patterns, EOG data, and cognitive load, we have tocompare the measurements of subjects in situations demanding varying levels of cognitive engagement. To induce the required states we used video clips of two different categories, with the first consisting of Hollywood movie
trailers. Trailers are traditionally produced and edited forthe purpose of drawing the viewer’s attention in a short period of time (usually about 2,5 minutes). Therefore they are
ideal for an experimental setup like this. Our participants
2961

Late-Breaking Work

CHI 2017, May 6–11, 2017, Denver, CO, USA

had to follow the storyline and pay as close attention to thecontents as possible. They were informed that they will be
asked questions concerning the contents of the trailer andthat we will count right and wrong answer. After each trailer,
we went through questions of varying difficulty regarding
all kinds of facts presented in the trailers, e.g. names of
production companies, publishing dates, spoken phrases,
and details such as titles of books only briefly shown. We
selected the official movie trailers for "Cloverfield", "Wild",and "The Theory of Everything". The second video category showed five minutes of an uncut seashore scene.
The video was not edited, had no story, and only its natural soundtrack. This allowed us to keep emotional triggers
or other possible attention grabbing stimuli away from theviewer. When watching this video, participants were asked
to relax or even meditate, if they preferred to.

Figure 2: Candidate wearing J!NSMEME watching a video while
facial temperature is logged

For a first experiment run, we recruited five university students between the ages of 20 and 35 of which two werefemale. All candidates had normal or corrected to normal
vision and were of different academic backgrounds. Before starting the actual tasks, we engaged each participant
for ten minutes in a light chat in order to allow their facial
temperature to adjust to the room climate. The room temperature was controlled by a digital wall thermostat and set
to 21◦ C. After this acclimatization period subjects were introduced to the coming tasks. The first three candidates
were initially watching the Hollywood trailers followed by theseashore sequence. The last two students were presented
with the videos in opposite order. After each of the trailers,participants were asked 15 questions from a questionnaire.
Even though we registered the number of correct and incorrect answers, the sole purpose of the questionnaire wasto motivate candidates to pay close attention to the trailer
content and obtain demographic information.

The participants’ EOG data was recorded using J!NS MEME
smart glasses equipped with three EOG electrodes for logging eye movement and eye blink. Facial temperatures
were recorded using a Seek Thermal XR camera at around
15 FPS during the whole study. The complete setup including a candidate wearing J!NS MEME can be seen in Figure 2. We sampled the temperature in the beginning of theexperiment before the first video was played to receive abaseline value. During the trailers we logged temperaturesin the middle and in the end of each trailer, and for the surveys we monitored the temperature three times, namely
before starting the survey, after half of the survey, and rightafter the last answer was given.

Evaluation
Two participants knew the films belonging to the trailers(Group A) and showed different temperature signatures
over the course of the experiment than the other three candidates (Group B), to whom all film trailers were new. We
logged the temperature changes of 11 ROIs on each candidate’s face (Figure 3). These ROIs were: forehead top(FT), forehead left (FL), forehead center (FC), foreheadright (FR), forehead bottom (FB), eye left (EL), nose top(NT), eye right (ER), cheek left (CL), nose bottom (NB), andcheek right (CR). The analysis of the thermography and
EOG data showed some interesting findings and trends.
We could identify three major areas on the face that show
almost identical results when compared to each other.
These areas were forehead and eyes (A1), cheeks (A2),and Nose (A3). This means that the comparison of datafrom any ROI of the forehead to an ROI of the cheeks ornose will result in similar trends of temperature changes,
which perfectly corresponds to the vascular anatomy of theface. A1 are supplied by ophthalmic artery. A2 are suppliedby the facial and infraorbital artery, and A3 are supplied bybranches of both, facial and ophthalmic arteries, but since2962

Late-Breaking Work

CHI 2017, May 6–11, 2017, Denver, CO, USA

the nose acts as a heatsink and because of increased respiration during the interview A3 temperature change patterns are significantly different from A1 and A2. In order toexplain this in more detail, we picked two significant sets ofdata in the following that utilize temperatures from ROIs thatare in reach of contactless sensors attached to eyewear,
and therefore of particular interest for us.
The most striking development was presented by the temperature gradient between the ROIs "cheeks" (average of
CL and CR) and the FC area (between the eyebrows andabove the bridge) of Group B. All temperatures provided are
relative differences between two ROIs, and no absolutes.
During the trailer presentation, the average cheek temperature rose by 1.49◦ C and dropped by an average of 0.43◦ C
within 20 seconds after the trailer finished in comparison to
FC. The temperature then rose gradually from the beginning until the end of each survey by an average of 0.99◦ C(beginning), 1.05◦ C (half way), and 1.14◦ C (end). The maximum drop in temperature we measured for Group B within
the first 20 seconds after a trailer finished was -1.32◦ C.
The survey average temperature was 1.14◦ C higher on thecheeks compared than on the central forehead. The average temperature of the cheeks decreased by 0.35◦ C whencomparing video and survey periods. The forehead temperature fluctuates insignificantly around the average temperature. In comparison to Group B, Group A’s readings show
only minimal variations without neither any stable trends
nor significant changes. This shows a first clear trend forcognitive engagement causing measurable temperature
differences between the cheek region and FC.
In a second round of analysis, we compared the temperature changes of NT and FC. For Group B the average
NT temperature while watching videos was 1.48◦ C lower
than that of FC. During the surveys the NT temperature

Figure 3: Candidate wearing J!NS MEME watching a video while
facial temperature is logged

decreased even further to -1.9◦ C in average compared tothe temperature measured on FC. As in the former set,
we found significant temperature drops of approximately
0.32◦ C, with one candidate where the temperature dropped
0.7◦ C within 10-20 seconds after the videos stopped playing and between seven and 10 seconds into the survey
questions. When comparing the NT temperature during thevideo presentation with the temperature during the surveys,
we find an average decrease of 0.42◦ C. This is the samedevelopment as in the first example. As with the cheeks,
NT suddenly got colder during the surveys whereas the FC
temperature did not show any significant trends. In conclusion, we can say that the temperatures in A2 and A3
decrease during the Q&A while temperatures in A1 slightly
fluctuate compared to the average facial temperature, but
do not present any identifiable trends. Temperatures forGroup A followed the same trends, but the differences were
not as significant as for Group B.
2963

Late-Breaking Work

CHI 2017, May 6–11, 2017, Denver, CO, USA

The logged EOG data was used to identify eye blinks ofcandidates watching the trailers, answering the survey
questions, and watching the unedited video. In Figure 4 wecan see one example for each measurement. It shows a six
second extract of the recorded eye blinks. The height of thepeaks in the three graphs describes the force and the width
the speed of the eye movement that can be monitored for
identifying eye blinks. Therefore, we can identify fewer quick
blinks during the trailer presentation, whereas blinks wereslow and strong while candidates watched the seashore
video. According to Nakano, delayed, quick eye blinks area sign of higher attention [10]. In comparison a slower blinkrate indicates relaxation and fatigue, in our case inducedby the seashore video. As can be seen in the center image of Figure 4, our blink rate speeds up significantly while
speaking [3]. The average blink rates we measured wereas follows: 1/sec while watching trailers, 2/sec during thesurvey, and 0,5/sec in reaction to the seashore video.

Conclusion

Figure 4: EOG data of logged eye
blinks for 6 seconds during Top:
Trailer, Center: Survey, Bottom:
Seashore VideoThe clear EOG results in combination with the outcomes of
the thermography analysis show a definite relation between
cognitive efforts of various degrees with eye blink rate andfacial temperature changes. Even though the temperaturechanges are not a novel finding, we could discover new
ROIs that are of significant meaning for the developmentof unobtrusive eyewear that can help us track our attention and therefore make it measurable in every day situations. Precisely, attached to the bridge of the glasses wewill use sensors pointing upwards and downwards for measuring the temperature changes in FC and NT. Additional
placements could be the rim of the glasses pointing at thewearer’s cheeks and eyes. Our setup was designed exclusively with low cost off-the-shelf products. We believe that
we could show the potential of such a design and strongly
trust in its unique potential for future studies and our goalof developing an open eyewear platform for the better management of attentional resources in every day situations.

Future WorkAs mentioned before, this study describes a first step towards a smart eyewear platform capable of tracking cognitive processes, such as attention. Our next step will be thecreation of a physical prototype of eyewear equipped withcontactless IR temperature sensors to be used in a reproduction and extension of this study. We will furthermore add
areas on and around the ears and sides of the head to our
ROIs for future temperature measurements.
Additionally, we are considering the implementation of facial
Galvanic Skin Response (GSR) measurements in futurestudies. We are not aware of any attempts that utilize GSR
on smart eyewear for cognitive activity tracking. The skinconductance is influenced by the amount of sweat excretedby the skin, thus GSR can be a good indicator of excitement/stress or relaxation levels. We are working on using
the electrodes of J!NS MEME for measuring facial GSR.

Acknowledgments
This research is supported by JST Presto.

REFERENCES
1. Reginald B. Adams, Nicholas O. Rule, Robert G.
Franklin, Elsie Wang, Michael T. Stevenson, Sakiko
Yoshikawa, Mitsue Nomura, Wataru Sato, Kestutis
Kveraga, and Nalini Ambady. 2009. Cross-cultural
Reading the Mind in the Eyes: An fMRI Investigation.
Journal of Cognitive Neuroscience 22, 1 (2017/01/10
2009), 97–108. DOI:
http://dx.doi.org/10.1162/jocn.2009.21187
2. Oliver Amft, Florian Wahl, Shoya Ishimaru, and Kai
Kunze. 2015. Making Regular Eyeglasses Smart. IEEE
2964

Late-Breaking Work

CHI 2017, May 6–11, 2017, Denver, CO, USA

Pervasive Computing 14, 3 (July 2015), 32–43. DOI:
http://dx.doi.org/10.1109/MPRV.2015.60
3. Anna R. Bentivoglio, Susan B. Bressman, Emanuele
Cassetta, Donatella Carretta, Pietroand Tonali, andAlberto Albanese. 1997. Analysis of Blink Rate
Patterns in Normal Subjects. Movement Disorders(1997).
4. Marc Germain, Michel Jobin, and Michel Cabanac.
1987. The effect of face fanning during recovery fromexercise hyperthermia. Journal of PhysiologicalPharmacology 65 (1987), 87–91.
5. Curtis S. Ikehara and Martha E. Crosby. 2005.
Assessing Cognitive Load with Physiological Sensors.
In Proceedings of the 38th Annual Hawaii International
Conference on System Sciences. 295a–295a. DOI:
http://dx.doi.org/10.1109/HICSS.2005.103
6. Shoya Ishimaru, Kai Kunze, Yuji Uema, Koichi Kise,
Masahiko Inami, and Katsuma Tanaka. 2014. Smarter
Eyewear: Using Commercial EOG Glasses for Activity
Recognition. In Proceedings of the 2014 ACM
International Joint Conference on Pervasive andUbiquitous Computing: Adjunct Publication (UbiComp’14 Adjunct). ACM, New York, NY, USA, 239–242. DOI:
http://dx.doi.org/10.1145/2638728.2638795
7. J. Kang, John A. McGinley, Gary Mcfadyen, and Kari
Babski-Reeves. 2006. Determining learning level andeffective training times. In Proceedings of the 25th
Army Science Conference (V43).
8. Thomas V. McCaffrey, R. D. McCook, and Robert D.
Wurster. 1975. Effect of head skin temperature on
tympanic and oral temperature in man. Journal ofApplied Physiology 39 (1975), 114–118.

9. Merriam-Webster.com. 2017. attention. (2017).
Retrieved January 3, 2017 from https://www.merriam-webster.com/dictionary/attention.
10. Tamami Nakano, Yoshiharu Yamamoto, Keiichi Kitajo,
Toshimitsu Takahashi, and Shigeru Kitazawa. 2009.
Synchronization of spontaneous eyeblinks while
viewing video stories. Proceedings of the Royal Societyof London B: Biological Sciences 276, 1673 (2009),
3635–3644. DOI:
http://dx.doi.org/10.1098/rspb.2009.0828
11. US; Lions clubs International Foundation National
Eye Institute. 2007. 2005 survey of public knowledge,
attitudes, and practices related to eye health anddisease. (2007).
12. Greg J. Siegle, Naho Ichikawa, and Stuart R.
Steinhauer. 2008. Blink before and after you think:
blinks occur prior to and following cognitive load
indexed by pupillary responses. Psychophysiology 45,
5 (2008), 679–687.
13. A. L. Sukstanskii and D. A. Yablonskiy. 2007.
Theoretical limits on brain cooling by external headcooling devices. European Journal of Applied
Physiology 101, 1 (2007), 41–49. DOI:
http://dx.doi.org/10.1007/s00421-007-0452-5
14. Benjamin Tag, Junichi Shimizu, Chi Zhang, Kai Kunze,
Naohisa Ohta, and Kazunori Sugiura. 2016. In the Eyeof the Beholder: The Impact of Frame Rate on Human
Eye Blink. In Proceedings of the 2016 CHI Conference
Extended Abstracts on Human Factors in Computing
Systems (CHI EA ’16). ACM, New York, NY, USA,2321–2327. DOI:
http://dx.doi.org/10.1145/2851581.2892449

2965

Late-Breaking Work

CHI 2017, May 6–11, 2017, Denver, CO, USA

15. Huan Wang, Bonnie Wang, Kieran P. Normoyle, KevinJackson, Kevin Spitler, Matthew F. Sharrock, Claire M.
Miller, Catherine Best, Daniel Llano, and Rose Du.
2014. Brain temperature and its fundamental
properties: a review for clinical neuroscientists.
Frontiers in Neuroscience 8 (2014), 307. DOI:
http://dx.doi.org/10.3389/fnins.2014.00307

16. YouGov. 2011. 2011 survey, N=1200. (2011).
https://today.yougov.com/news/2011/05/05/
brother-do-you-have-time/
17. Robert B. Zajonc, Stephanie Murphy, and Marita
Inglehart. 1989. Feeling and facial efference:
implications of the vascular theory of emotion. Journalof Applied Physiology 96, 3 (1989), 395–416.

2966