My Reading Life – Towards Utilizing
Eyetracking on Unmodified Tablets
and Phones
Kai Kunze

Yuzuko Utsumi

Graduate School of Engineering
Osaka Prefecture University
1-1 Gakuen-cho, Naka, Sakai,
Osaka, Japankunze@m.cs.osakafu-u.ac.jp

Graduate School of Engineering
Osaka Prefecture University
1-1 Gakuen-cho, Naka, Sakai,
Osaka, Japanutsumi@cs.osakafu-u.ac.jp

Shoya Ishimaru

Koichi Kise

Graduate School of Engineering
Osaka Prefecture University
1-1 Gakuen-cho, Naka, Sakai,Osaka, Japan
ishumaru@m.cs.osakafu-u.ac.jp

Graduate School of Engineering
Osaka Prefecture University
1-1 Gakuen-cho, Naka, Sakai,Osaka, Japan
kise@cs.osakafu-u.ac.jp

Abstract
As reading is an integral part of our knowledge lives, weshould know more about our reading activities. This
paper introduces a reading application for smart phone
and tablets that aims at giving user more quantified
information about their reading habits. We present ourwork towards building an open library for eye tracking on
unmodified tablets and smart phones to support some of
the applications advanced functionality. We implemented
already several eye tracking algorithms from previouswork, unfortunately all seem not to be robust enough forour application case. We give an overview about ourchallenges and potential solutions.

Author Keywords
reading, knowledge, eye tracking, mobile devices, eye gaze

ACM Classification Keywords
H.5.2 [User Interfaces]: Input devices and strategies;
H.5.m [Information interfaces and presentation (e.g.,HCI)]: Miscellaneous.
Permission to make digital or hard copies of part or all of this work forpersonal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. Copyrights for third-party
components of this work must be honored. For all other uses, contact theowner/author(s). Copyright is held by the author/owner(s).
UbiComp’13 Adjunct, September 8–12, 2013, Zurich, Switzerland.
ACM 978-1-4503-2215-7/13/09.
http://dx.doi.org/10.1145/2494091.2494179

Introduction
The more somebody reads the higher seems to be
language skills and general knowledge [6]. People canincrease their physical fitness and reduce their weight bystep counting and food logging [3]. So it seems simple

tracking of activities raises awareness and leads tohealthier habits. We therefore wonder if we can give
people tools to better track their reading and thus
improve their cognitive life. Unfortunately there are very
few e↵orts to recognize reading activities in realistic
environments [5, 14, 12].
We are driven by the research question, how to monitor
and improve knowledge acquisition tasks with a special
focus on reading.

Figure 1: Article view of the“My Reading Life” application(auto-scrolling can be done usingeye gaze).

The contributions in this paper are outlined as follows.
(1) We introduce a prototype of a smart phone
application called “My Reading Life”, that takes articles
from a popular read later web service and provides you
with summary of your reading using statistics over UIinteractions and our eye gaze library. (2) We show how we
obtain several statistics about the users reading habits(using eye position and head angle to estimate if the user
is looking on the device), discuss challenges and potentialsolutions. (3) We use a combination of standard computer
vision techniques for gaze estimation to implement several
eye tracking methods on an unmodified tablet and smartphone ( iPad 4 and iPhone 5) with the goal to provide anopen standard eye gaze library for researchers anddevelopers. Unfortunately, the performance depends so far
highly on the calibration, the user and lighting conditions.

Approach

Figure 2: Sample summary
statistics: average number ofarticles read, the genre, reading
speed, popular sources,
comparison with friends, andreading locations.

The most profound, yet a very difficult and long way to
track reading habits is to equip the user with a dedicateddevice (e.g. eye tracker and scene camera) to record all
reading interactions from the users point of view.
In this paper we go a di↵erent route, we want to use
digital devices to track the users reading habits. More andmore reading is done on them and they have some uniqueadvantages, as we can easily obtain additional meta
information about the reading material at hand without
using expensive machine learning or computer vision
techniques (e.g. the text without using OCR). Weenvision a Reading Service similar to a Location Servicebuilt-in to all kinds of digital devices, enabling the user totrack their reading.

Implementation
To see if we are able to provide users with enoughquantifiable information to change their reading habits,
we implemented a prototype application on the iOSplatform. To supply it with reading material, we use a
popular read later service http://readability.com forwebsite articles. We import the reading list and let users
view the articles in a standard view, see Fig 1.
Tracking Reading Habits
We give the user feedback over the following statistics:
the number of articles read, the average reading speed,
the genre of the article, the location the user read thearticle, the orientation of the device, as well as the urlsource of the article, a comparison with the users friends
according to articles read this week and speed (seeFig. 2). The first prototype uses the head angle and eye
positions to estimate if the user is looking on the screen.
To determine both we use the in iOS built-in facedetection and a Haar Cascade for the eyes (a standard
computer vision technique see [16]). If the user seems tolook at the screen for a longer period of time and scrolls
to the end of the article not to fast (estimated with athreshold at around 700 words per minute), we count thearticle as read and record the approximate reading speed.
The genre of the article is determined using Latent
Semantic Mapping and some default categories, so far:culture, science, sports, economics, politics. These can be

Figure 3: One of the 5calibration points the user needs
to focus on. 4 are in the cornersof the text field, the fifth is in themiddle of the screen.

Figure 4: Successful eye gazeestimation after calibration on
the iPad. The user reads the firstword in the first line.

easily adjusted and the user can provide their own over“tagging”. The comparison with friends is so far static,
due to the small user count, yet we want to include aFacebook/Twitter connection.

We believe that some problems can be solved by using thedevice orientation obtained from motion sensors in thedevice and by combining several approaches presented byHansen et.al. [9].

Towards Robust Eyetracking
The current implementation of “reading detection” is very
limited. As we want to improve it, we implemented
several standard eye tracking and gaze estimation
techniques from computer vision. We started with anappearance based approach presented by Holland et.
al. [11] (code is available under GPL3) and reachedsimilar accuracies as mentioned in their paper. Yet, thismethod is not fit for our problem, as it requires a longtraining phase and is highly dependent on the user. Sofar, a hybrid approach using a combination of existing
techniques works best, described in the following. We
implemented a shape-based approach using model fitting
method based on elliptical shape models [10], detecting
eye corners and iris shape by first using face and eye
detection over a Haar Cascade [16]. This works faster(computationally less intensive), needs fewer training
samples and seems more robust towards di↵erent users(Fig. 4 and 5). The method requires a five pointcalibration (see Fig. 3) and the user should not change
the orientation of the device after calibration is done.

In addition to better working reading detection, robust eye
gaze estimation and tracking can enable completely newapplications. It’s shown that eye gaze is related toknowledge and expertise, also the use of eye gestures ispossible [13, 4].

However, we still have some calibration problems andfaulty detections, if the wrong parameters are used (seeFig. 6 and Fig. 7 for an example). All methods are very
dependent on the calibration phase, lighting conditions
and how the user holds the device. Interestingly, a user
reading on the device shows a very characteristic patterneven when calibration has failed. Therefore we are
currently investigating if we can detect reading despite
not being able to get a stable gaze estimate.

There is some work to foster better reading habits inchildren [7]. Yet, to our knowledge there is so far no e↵ort
towards quantified feedback for adults.

Related Work
The closest to our work is an research by Holland et. al.
using an appearance-based approach to estimate a user’s
gaze, training a neural network on di↵erent eye
positions [11]. Also there are several papers that use
commodity webcams for eye tracking [11]. Hansen et. al.
give a great reference and detailed overview aboutdi↵erent implementations of eye gaze estimation andtracking [9, 10].
Concerning reading, there are a few papers that try todetect skimming versus reading with dedicatedeyetrackers [8, 2]. Eye tracking while reading can also be
used to provide summaries of documents or to find which
words a user finds relevant [15].

Conclusion and Future WorkWe will explore further how to implement reading
detection and a more stable eye tracking. A more
important goal is to get first user feedback on the“reading life” application to see what type of inferenceand statistic can spur better reading habits and apotential increase in reading.
We are working on making the gaze library open available,
currently we focus on iOS, yet we also plan to support
Android. Maybe it is possible to implement some of theadvanced interaction techniques on commercial, mobile
devices outlined by Biedert et. al. using dedicatedeyetrackers [1].

Acknowledgements
Figure 5: Successful eye gazeestimation after calibration onthe iPhone. User reads the lastword in the 3rd line.

Figure 6: Relatively good irisestimation.

Figure 7: Bad parameters for irisestimation.

This work was supported in part by the CREST projectfrom Japan Society for the Promotion of Science (JSPS).

References[1] Biedert, R., Buscher, G., and Dengel, A. The
eyebook–using eye tracking to enhance the readingexperience. Informatik-Spektrum 33, 3 (2010),272–281.
[2] Biedert, R., Hees, J., Dengel, A., and Buscher, G. A
robust realtime reading-skimming classifier. In Proc.
ETRA 2012 (2012), 123–130.
[3] Bravata, D. M., and Smith-Spangler. Using
pedometers to increase physical activity and improvehealth. The journal of the American MedicalAssociation 298, 19 (2007), 2296–2304.
[4] Bulling, A., and Roggen, D. Recognition of visualmemory recall processes using eye movement
analysis. In Proc. UbiComp 2011 (2011), 455–464.
[5] Bulling, A., Ward, J. A., and Gellersen, H.
Multimodal Recognition of Reading Activity inTransit Using Body-Worn Sensors. ACM Trans. on
Applied Perception 9, 1 (2012), 2:1–2:21.
[6] Cunningham, A., and Stanovich, K. What reading
does for the mind. Journal of Direct Instruction 1, 2(2001), 137–149.

[7] Doiron, R. Using e-books and e-readers to promotereading in school libraries: Lessons from the field,2011.
[8] Duggan, G. B., and Payne, S. J. Skim reading by
satisficing: evidence from eye tracking. In Proc. ofCHI 2011 (2011), 1141–1150.
[9] Hansen, D. W., and Ji, Q. In the eye of the beholder:
A survey of models for eyes and gaze. PatternAnalysis and Machine Intelligence, IEEE Transactions
on 32, 3 (2010), 478–500.
[10] Hansen, D. W., and Pece, A. E. Eye tracking in thewild. Computer Vision and Image Understanding 98,1 (2005), 155–181.
[11] Holland, C., and Komogortsev, O. Eye tracking on
unmodified common tablets: Challenges andsolutions. In Proceedings of the Symposium on Eye
Tracking Research and Applications, ACM (2012),
277–280.
[12] Kunze, K., Bulling, A., Utsumi, Y., Yuki, S., andKise, K. I know what you are reading – recognitionof document types using mobile eye tracking. In in
Proc. of ISWC 2013 (2013).
[13] Kunze, K., Kawaichi, H., Yoshimura, K., and Kise,K. Towards inferring language expertise using eye
tracking. In Ext. Abs. CHI 2013 (2013), 4015–4021.
[14] Kunze, K., Kawaichi, H., Yoshimura, K., and Kise,K. The wordmeter – estimating the number of words
read using document image retrieval and mobile eye
tracking. In Proc. ICDAR 2013 (2013).
[15] Loboda, T. D., Brusilovsky, P., and Brunstein, J.
Inferring word relevance from eye-movements ofreaders. In Proc. of IUI ’11 (2011), 175–184.
[16] Schmidt, A., and Kasiński, A. The performance ofthe haar cascade classifiers applied to the face andeyes detection. In Computer Recognition Systems 2.
Springer, 2007, 816–823.