UBICOMP/ISWC ’16 ADJUNCT, SEPTEMBER 12-16, 2016, HEIDELBERG, GERMANY

Reading Interventions - Tracking
Reading State and Designing
Interventions
Shoya Ishimaru

Kai Kunze

Abstract

Graduate School of Engineering
Osaka Prefecture University
Sakai, Osaka, Japanishimaru@m.cs.osakafu-u.ac.jp

Graduate School of Media Design
Keio University
Yokohama, 223-8526 Japan
kai@kmd.keio.ac.jp

Tilman Dingler
University of Stuttgart
Institute for Visualization andInteractive Systems
tilman.dingler@vis.unistuttgart.de

Koichi Kise

The quantified-self movement has brought us a multitudeof tracking devices mostly focused on physical activities.
But healthy user habits go beyond the physical realm, suchas cognitive activities, learning and reading. For installing
healthy reading habits we built a wordometer using smart
eyewear for tracking online and offline reading activities
throughout the day. Besides providing reading statistics atthe end of the day we propose a number of real-time interventions to support reading in-situ: adjustable text thattakes into account users’ current cognitive state, such asattention span, visual fatigue and comprehension levels. In
this paper we report insights from our in-the-wild reading
tracker and explore various intervention techniques with thegoal of supporting user attention, focus, and comprehension in reading tasks.

Graduate School of Engineering
Osaka Prefecture University
Sakai, Osaka, Japan
kise@cs.osakafu-u.ac.jp

Andreas Dengel
DFKI
Kaiserslautern, Germany
andreas.dengel@dfki.deAuthor Keywords
reading; context-awareness; quantified mind;
Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the Owner/Author. Copyright is held by the owner/author(s).
UbiComp/ISWC’16 Adjunct, September 12-16, 2016, Heidelberg, Germany
ACM 978-1-4503-4462-3/16/09.
http://dx.doi.org/10.1145/2968219.2968271

1759

ACM Classification Keywords
H.5.m [Information interfaces and presentation]: misc.

Introduction"Reading is thinking with someone else’s head instead ofone’s own" Albert Schopenhauer once stated. It opens up
new perspectives by taking the reader on a mental journey

UBICOMP/ISWC ’16 ADJUNCT, SEPTEMBER 12-16, 2016, HEIDELBERG, GERMANY

for information gain and pleasure. Reading informs, builds
vocabulary and even benefits the ability to judge and argue[17]. Especially in today’s information age, the skill to efficiently consume, absorb, and evaluate information is morecrucial than ever. Throughout our day we constantly engage with written text, be it in the morning newspaper, on
car displays, billboards, our desktop computer, cellphone, or
smartwatch. It would appear that by instilling healthy reading habits we can help readers be more effective throughout
their day.
Acquiring or changing habits is one of the claimed goals of
the quantified-self movement. Fitness and sleep trackers,
for example, allow us to gain insights into our daily habits
in order to be able to take action against unhealthy behavior. However, most of the commercial products and also
most of the efforts of the research community have so far
focussed on tracking physical activities. Only a handful of
proponents have suggested the tracking of cognitive acitvities [11], which can entail attention, mental fatigue, or
stress levels. In our work we focus on using eye movements
to track reading behavior throughout the day and deriving
attention levels during reading.
Schopenhauer’s statement above also contains the notionof guiding the reader. By being able to track reading activities throughout the day, we can design systems to bettertake into account the reader’s current level of attention or
fatigue. Reading interfaces can process this information in
real-time and adjust their presentation to meet the reader’scurrent cognitive state, such as her general receptiveness.
By doing so we create in-situ interventions with the goal ofincreasing reading efficiency. In this paper we describe ourapproach to track reading throughout the day and acrossdifferent mediums and start exploring the notion of readinginterventions. We believe that interventions can be staged

1760

Figure 1: A screen shot image of web based dashboard to lookback volumes of reading in everyday.

in such a way that we can
1. facilitate reading tasks,
2. support associated cognitive processes, such ascomprehension and recall, and
3. instill healthy reading habits.

Related Work
The strong relationship between reading and eye movements is very well explored in cognitive science and psychology [16, 8]. For example, Kligel et al. investigate correlations of eye fixations with cognitive tasks related to reading [10]. Rayner provides a good summary of eye tracking
research [15].

SESSION: EYEWEAR

3"Point(Electrography
Ba;eryMo4on(Sensors
Bluetooth(LE(etc.

In a series of works, Biedert et al. studied ways to enhance
the reading experience of the user. They presented EyeBook [1] and Text 2.0 [2] a reading interface that observe
which part of the text is currently being read by the user
and that generate appropriate effects (e.g. playing sounds).
However, they don’t evaluate what suitable interventions
are to increase users enjoyment, comprehension or attention. Xu et al. apply eye movement analyzes for document
summaries, yet the environment is very controlled, e.g. theusers need to rest their chin on a support when performing
the reading task [18].
Several researchers also focus on increasing reading speed
minimizing the loss of comprehension [4, 9]. To this end, we
evaluated several speed reading techniques and stimuli [5].

Figure 2: J!NS MEME hardware
on top and the positions of threeelectrodes and formulas tocalculate EOG vertical andhorizontal component.

There are also some efforts to infer the users expertise, language skill and other higher level cognitive activities using
eye tracking [12, 14, 3, 6, 7]. Most of the research focusing
on second language learners or infants as improvements
can be easier tracked using indirect measurements (questionnaires etc.). Our current investigation into reading interventions continues our strand of research looking at how toquantify reading [13].

The Wordometer
One of the metrics to understanding reading habits is volume of reading. We quantify the number of read words as
the volume of reading and show them on the visualization
as shown in Figure 1. Each bar represents the number ofwords users read in one minute. From the visualization,
users can understand how much they read and when theyespecially focused to read in segments.

1761

Figure 3: Overview of EOG sensor signal in one-minute recording

Figure 4: EOG sensor signal during reading. Circle and triangle
markers are forward/backward saccade detection’s outputs.

Word Counting Algorithm
The word counting algorithm consists of three processes,
obtaining a user’s eye gaze, detecting forward/backwardsaccades, and estimating the number of words he/she read.
To obtain a user’s eye gaze, we use J!NS MEME whichis connected to a smartphone with Bluetooth low energy.
These are affordable technologies and good for long-term
recordings. As shown in Figure 2, one reference and two
active electrodes are equipped on J!NS MEME. EOG vertical component is calculated as an average of two electrodes, and EOG horizontal component is calculated as adifference between them.
Figure 3 shows an overview of EOG horizontal component
in one-minute recording that includes reading activity. Negative values represent eye movement right to left, and positive values represent eye movement left to right. RegularUBICOMP/ISWC ’16 ADJUNCT, SEPTEMBER 12-16, 2016, HEIDELBERG, GERMANY

patterns of eye movement will appear during reading activity because of line breaks.

Figure 5: A user reading on atablet with J!NS MEME.

Subject
a
b
c
d
e
Ave.

Error rate
14 %
24 %
15 %
15 %
12 %
16 %

STD
1.6 %
2.1 %
1.8 %
1.7 %
1.4 %
1.7 %

Table 1: Estimation results on
user-dependent approach

We detect these saccades to forward and backward fromsensor signals as shown in Figure 4. For forward-saccade
detection, we filter the sensor signal with median filter toremove noises. After filtering, we apply peak detection algorithm to detect forward-saccades. Backward-saccade is detected if the sensor value is lower than a threshold which is
decided dynamically. The threshold is calculated as the difference between mean value and variance value of sensor
values in a small window (the window size is one second).
The number of words a user read is estimated by linear
support vector regression. Four features are calculated
for the regression, the total count of forward-saccades,mean EOG signal value of forward-saccades, the total
count of backward-saccades, and mean EOG signal valueof backward-saccades.
Evaluation
As an initial evaluation before long-term recording, we evaluated our word counting algorithm with a dataset on fixed
condition. We recruited 5 participants to read English essays on iPad wearing J!NS MEME (see Figure 5). Everysubject read 38 paragraphs during the recording. So thetotal amount of paragraph in the dataset was 190 (minimum: 27, maximum: 120, average: 60 words in one paragraph). The sampling frequency of EOG signal from J!NSMEME was 11 Hz. Training and testing were done on bothuser-dependent and user-independent approach. On anuser-dependent approach, data from each subject are usedas training for his/her reading. On the user-independent
approach, all data from all subjects are used as training.
Evaluations were done with 10-fold cross validation.

1762

Table 1 shows the estimation errors and standard deviations on user-dependent approach. The average error offive participants was 16% (9.6 words in 60 word’s paragraph). The estimation error on user-independent approach
was 18% with 1.7% STD. Reading recognition and quantification is the first step towards building applications dealing
with this type of data. One of the focuses of our future work
now is the creation of reading interventions based on theassumption that reading is tracked in real-time and underlying text is dynamic, such as on tablets.

Reading Interventions
There is a multitude of information the tracking of eye movements can give us about the cognitive state of readers. For
example, blinking rates give us insight into visual fatigue
levels [15]. So as readers are getting more tired their eye
blinking rate rises. Such fatigue is often temporary and canbe met with a distraction task, such as standing up, putting
down the book and stretch for a few minutes. Also gettingsome fresh air or water could be one of the recommendations to regain the ability to concentrate. In the following wewill describe potential interventions:
For one, we can aim at increasing comprehension: nudge
readers to pause after stretches of reading to reflect onwhat has just been read. Such behavior further fosters later
recall of that content.
On similar note, pupil dilation corresponds to engagement
and attention levels. This can be used in two ways: 1) assummary metric where text that was read with high attention level might be summarized with more detail than textthat was comparatively less intensely attended to (assuming that attention correlates with the reader’s interest in thecontent) and 2) as feedback mechanism to authors andpublishers to refine their content according to readers’ at-SESSION: EYEWEAR

tention (this could also comprise ads being priced higher toreach people with high attentiveness).
Regressions - i.e. re-reading of words, sentences, or entire
passages - account for about 15% of the reading time. Although being helpful for comprehension, it often turns intoa habit that slows down the reading process. Simply being aware of regressions through a feedback mode could
encourage readers to keep going forward through text andthereby increasing their reading speed. In-situ reading stimuli can be applied in order to encourage users to keep orincrease their reading pace [5].

Experiments
To test and assess the effects on reading we are planninga series of user studies based on the reading intervention
techniques presented. Therefore, we are currently developing a synchronization feature that pairs the eye tracking
data from J!NS MEME glasses with the reading application
running on a tablet device. That way we can stage interventions visually, through haptic feedback as well as throughaudio cues. In lab studies we will collect subjective user
feedback and reading metrics to single out useful interventions. For long-term assessment we will hand out devices
to study participants over longer periods of time where wewill collect data in the wild about people’s reading habits
throughout the day, but also about their experience withreading interventions when being used in combination withour complementing app.

Conclusion
In this paper we presented an extension of our initial Wordometer technology (detecting how many words a user
reads) to a wearable, unobtrusive eye wear (J!NS MEME)
and outlined a couple of reading interventions suitable forthis setup. In the intermediate next steps we need to ex-1763tend our experimental setup to a larger, controlled set toevaluate our basic technology more. As outlined in the previous section we plan to study reading in the wild to first of
all understand what are healthy reading habits and then to
improve them.

References[1] Ralf Biedert, Georg Buscher, and Andreas Dengel.
2010a. The eyebook–using eye tracking to enhance
the reading experience. Informatik-Spektrum 33, 3(2010), 272–281.
[2] Ralf Biedert, Georg Buscher, Sven Schwarz, ManuelMöller, Andreas Dengel, and Thomas Lottermann.
2010b. The text 2.0 framework: writing web-basedgaze-controlled realtime applications quickly and easily. In Proceedings of the 2010 workshop on Eye gazein intelligent human machine interaction. ACM, 114–
117.
[3] Andreas Bulling and Thorsten O Zander. 2014.
Cognition-Aware Computing. Pervasive Computing,IEEE 13, 3 (2014), 80–83.
[4] Oscar de Bruijn and Robert Spence. 2000. Rapid
serial visual presentation: a space-time trade-off in information presentation. In Advanced visual interfaces.
189–192.
[5] Tilman Dingler, Alireza Sahami Shirazi, Kai Kunze,and Albrecht Schmidt. 2015. Assessment of Stimuli
for Supporting Speed Reading on Electronic Devices.
In Proceedings of the 6th Augmented Human International Conference (AH ’15). ACM, New York, NY,
USA, 117–124. DOI:http://dx.doi.org/10.1145/2735711.
2735796[6] Tamara van Gog, Liesbeth Kester, Fleurie Nievelstein,
Bas Giesbers, and Fred Paas. 2009. Uncovering cognitive processes: Different techniques that can con-

UBICOMP/ISWC ’16 ADJUNCT, SEPTEMBER 12-16, 2016, HEIDELBERG, GERMANY

tribute to cognitive load research and instruction. Computers in Human Behavior 25, 2 (2009), 325–331.
[7] Shamsi T Iqbal, Xianjun Sam Zheng, and Brian P Bailey. 2004. Task-evoked pupillary response to mental
workload in human-computer interaction. In CHI’04
extended abstracts on Human factors in computing
systems. ACM, 1477–1480.
[8] David E Irwin. 2004. Fixation location and fixation duration as indices of cognitive processing. The interfaceof language, vision, and action: Eye movements andthe visual world (2004), 105–134.
[9] Marcel A Just, Patricia A Carpenter, and Jacqueline D
Woolley. 1982. Paradigms and processes in reading
comprehension. Journal of experimental psychology:
General 111, 2 (1982), 228.
[10] Reinhold Kliegl, Antje Nuthmann, and Ralf Engbert.
2006. Tracking the mind during reading: the influenceof past, present, and future words on fixation durations. Journal of Experimental Psychology: General;
Journal of Experimental Psychology: General 135, 1(2006), 12.
[11] Kai Kunze, Masakazu Iwamura, Koichi Kise, Seiichi Uchida, and Shinichiro Omachi. 2013a. Activity
Recognition for the Mind: Toward a Cognitive "Quantified Self". In Computer, Vol. 45. 98–101.
[12] K. Kunze, H. Kawaichi, K. Yoshimura, and K. Kise.
2013b. Towards inferring language expertise using eye

1764tracking. In Ext. Abs. CHI 2013. 4015–4021.
[13] Kai Kunze, Katsutoshi Masai, Masahiko Inami, ÖmerSacakli, Marcus Liwicki, Andreas Dengel, Shoya Ishimaru, and Koichi Kise. 2015. Quantifying reading
habits: counting how many words you read. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM,
87–96.
[14] Pascual Martínez-Gómez and Akiko Aizawa. 2014.
Recognition of understanding level and language skill
using measurements of reading behavior. In Proceedings of the 19th international conference on Intelligent
User Interfaces. ACM, 95–104.
[15] Keith Rayner. 1998. Eye movements in reading andinformation processing: 20 years of research. Psychological bulletin 124, 3 (1998), 372.
[16] George D Spache. 1962. Is This a Breakthrough in
Reading? The Reading Teacher 15, 4 (1962), 258–
263.
[17] KE Stanovich and AE Cunningham. 1998. What reading does for the mind. American Education Journal(1998).
[18] Songhua Xu, Hao Jiang, and Francis C.M. Lau. 2009.
User-oriented document summarization throughvision-based eye-tracking. In Proc of IUI (IUI ’09). 7–16.