Poster: Electrooculography Dataset for Reading
Detection in the Wild
Shoya Ishimaru
German Research Center for
Artificial Intelligence (DFKI)
Shoya.Ishimaru@dfki.de

Takanori Maruichi
Osaka Prefecture University
maruichi@m.cs.osakafu-u.ac.jp

Koichi Kise
Osaka Prefecture University
kise@cs.osakafu-u.ac.jp

Manuel Landsmann
Osaka Prefecture University
m_landsman09@gmx.deAndreas Dengel
German Research Center for
Artificial Intelligence (DFKI)
Andreas.Dengel@dfki.de

ABSTRACT

1

Because of the diversity of document layouts and readingstyles, detecting reading activities in real life is a challenging
task compared to the detection in the laboratory setting. For
contributing to the implementation of robust reading detection algorithms, we introduce a dataset which contains 220
hours of sensor signals from JINS MEME electrooculography
glasses and corresponding ground truth activity labels. As abaseline study, we propose a statistical feature based reading
detection approach and evaluate it on the dataset.

Just as our bodies consist of what we eat, our minds areshaped by the information we obtain. In particular, written text is one of the most important information sourcesin our lives. Therefore, understanding and improving daily
reading habits provides several cognitive benefits, including
increased vocabulary and logical thinking [4].
For instance, as people are encouraged to be physically fit
by monitoring step counts, tracking the number of words
they read in a day has potential to motivate them to readmore. The idea of estimating the number of read words has
been implemented as Wordometer by using mobile eye tracking glasses [11], electrooculography glasses [6], and a remote
eye tracker [1]. However, these implementations were evaluated only in the laboratory setting. A finding from ourprevious in-the-wild study is that readings in real life occur in a variety of situations, and an estimator trained bywell-mannered reading data could not cover such natural
readings enough [5].
On the basis of several requests relating to the data, weconduct a large-scale recording again solving some issues
that appeared in the previous work. We utilize JINS MEME
(see Figure 1) as a sensing device. It is equipped with a threeelectrode electrooculography (EOG) sensor which measureseye movements and a six-axis internal measurement unit(IMU) which measures head and body movements. A form
factor and a long-life battery of the device are designed for inthe-wild studies. A wide variety of approaches have been proposed on it, including recognizing human activities [7], facialactions [13, 14], gaze gestures [15], and internal states [18].
The contributions of our work are two-fold: the EOG
dataset and our reading detection method. The dataset, software for the data recording, and sample codes are available
on our project repository1 . This study was conducted withthe permission of the Research Ethics Committee of the Graduate School of Engineering, Osaka Prefecture University.

CCS CONCEPTS• Computing methodologies → Activity recognition
and understanding.
KEYWORDS
Eyewear computing; Cognitive activity recognition; In-thewild study; Electrooculography; Internal measurement unit
ACM Reference Format:
Shoya Ishimaru, Takanori Maruichi, Manuel Landsmann, Koichi
Kise, and Andreas Dengel. 2019. Poster: Electrooculography Dataset
for Reading Detection in the Wild. In Adjunct Proceedings of the2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2019 International Symposium on Wearable
Computers (UbiComp/ISWC ’19 Adjunct), September 9–13, 2019, London, United Kingdom. ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/3341162.3343812
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. Copyrights for componentsof this work owned by others than ACM must be honored. Abstracting withcredit is permitted. To copy otherwise, or republish, to post on servers or toredistribute to lists, requires prior specific permission and/or a fee. Request
permissions from permissions@acm.org.
UbiComp/ISWC ’19 Adjunct, September 9–13, 2019, London, United Kingdom© 2019 Association for Computing Machinery.
ACM ISBN 978-1-4503-6869-8/19/09. . . $15.00
https://doi.org/10.1145/3341162.3343812

INTRODUCTION

1 https://github.com/shoya140/ubicomp2019-eog-dataset/UbiComp/ISWC ’19 Adjunct, September 9–13, 2019, London, United KingdomIshimaru et al.

(a) English(a) Device overview(b) Japanese horizontal (c) Japanese vertical(b) Sensor signals(d) Not reading

Figure 1: JINS MEME Electrooculography Glasses and sensor
signals visualized on our Android application

Figure 2: Examples of pictures taken by Narrative Clip while
reading or not reading in the wild2

sampling frequency was set to 100 Hz. Note that we modified kernel codes and built a customized Android operating
system for a stable Bluetooth Low Energy connection4 .
To support a ground truth labeling task, we provided Narrative Clip5 , a small life-logging camera which can be attached to clothes and takes a picture every 30 seconds. Examples of the pictures are shown in Figure 2. At the end ofeach day, every activity in the pictures was annotated by theuser into four categories: reading in English (EN), readingin Japanese written horizontally (JH), reading in Japanesewritten vertically (JV), and not reading (NR). The reason we
prepared such three labels for reading is that eye movements
should be affected by the language (native or non-native)
and writing style (vertical or horizontal). Even if characters
appeared in a picture, an activity which does not require
frequent line breaks was not categorized as reading in thisstudy (e.g., looking at a signboard while walking, reading
a comic, writing codes). In order to protect privacies, wecollected only activity labels and no pictures.
We recruited ten Japanese college students for two daysof data recording. To ensure collecting a minimum amountof valid reading behaviors, we asked them to try to perform
each activity (EN, JH, and JV) for at least one hour everyday. We held an initial briefing to explain the procedures
and the usage of the devices. In addition, all instructions
were written in a document and shared with participants.
Participants who completed the tasks received 10,000 JPY.
Table 1 shows an overview of the recording durations. In
total, our dataset contains 23 hours of English reading, 25
hours of Japanese horizontal reading, 25 hours of Japanesevertical reading, and 146 hours of other activities.

BACKGROUND AND RELATED WORK

As summarized in the survey by Lara and Labrador, many
physical activities (e.g., walking, running, cycling, sleeping)can be recognized by motion sensors on the body or a smartphone [12]. On the other hand, the recognition of reading
activity is considerably restricted because dynamic bodymovements could not be observed while reading.
One of the interesting approaches this problem is to measure eye movements. Steil and Bulling detected daily activities including readings by using mobile eye tracking
glasses [17]. Srivastava et al. recognized activities on a computer with several involving reading [16] and Kelton et al.
classified reading/skimming [9] by using a remote eye tracker
attached to a display. These sensors are good at measuring
gaze events (e.g., fixations and saccades). But we assume
that characteristic eye movements (e.g., frequent horizontal/vertical saccades) have enough potential for the detection.
EOG measures such eye movements from the corneoretinal standing potential that exists between the front andthe back of an eyeball. Traditional setups use four electrodes
around an eye [2, 3]. Recent sensing devices have three electrodes on each nose pad and the forehead to identify vertical/horizontal eye movements and blinks [8, 10].
3DATA RECORDING

We utilized JINS MEME ES_R2 and Android Nexus 5X for thedata recording. We developed an application3 which enables
a user to easily check sensor signals and start/stop recording data. Figure 1 shows a screen capture visualizing threeblinks and some vertical and horizontal eye movements. The
2 https://jins-meme.com/en/researchers/

4 https://shoya.io/posts/meme-android-connection/

3 https://memelogger.shoya.io/

5 http://getnarrative.com/

Poster: Electrooculography Dataset for Reading Detection in the Wild

UbiComp/ISWC ’19 Adjunct, September 9–13, 2019, London, United Kingdom

Table 1: Recording durations [minutes]

Table 2: Classification accuracies [%](a) Day 1
P1ENJHJVNR

P2P3P4P554 73 74 67 63
97 70 101 72 73
74 101 71 60 83
474 447 490 499 476

P6Condition
P7P8P9P1082 91 62 59
86 127 63 75
73 65 66 113379 204 538 476

499065494

P3P4P5P6P7P8P9P1096 61 61 74 115 67 55
74 117 53 64 60 53 58
75 67 60 116 72 74 73499 429 567 283 409 358 496585860307626875509

3246686768743634456966686946300200
EOG [μV]

EN 100JH 71
JV 89
NR 466

P2User-dependent

EN vs JH vs JV vs NR(EN + JH) vs JV vs NR(EN + JH + JV) vs NR
EN vs NR
JR vs NRJV vs NR
EN vs JH vs JV(b) Day 2
P1

User-independent

100
0
EOG_H
EOG_V

−100

40

READING DETECTION: BASELINE STUDY

We propose a detection method as a baseline. Characteristicsof the dataset found from an evaluation are also described.

51015Time [sec.]

202530Figure 3: EOG signals while reading an English document

ApproachTen features are calculated from one sample (30 seconds window of a data stream): means and variances of the two EOG
axes, variances of three accelerometer axes, and variancesof three gyroscope axes. Then we utilize a Support Vector
Machine to classify the samples. The radial basis function
kernel with hyper parameters C = 1 and дamma = 0.125
were selected experimentally and used for the classifier.
Experimental Conditions
We evaluated the proposed reading detection approach witha user-independent and a user-dependent training. For auser-independent training, we separated data into training
and testing in a a leave-one-participant-out cross-validationmanner. Data of nine participants were used for training aclassifier and data of the remaining one participant wereused for testing. For a user-dependent training, the classifier
was adapted to each participant by his/her data with leaveone-day-out cross-validation. One-day was used for training,
and another day was used for testing. Since the dataset isunbalanced and our purpose is to detect a minor class, weapplied under-sampling for the evaluation.
Results and Discussion
The classification results are shown in Table 2. Chance rates
are 50%, 33%, and 25% for two, three, and four-class classifications, respectively. One of the interesting findings fromthe results is that head and eye movements while reading JV

texts are relatively distinctive compared to EN and JH. Thismay be because the Japanese vertical writing style is often
used for well-formatted texts such as novels and newspapers.
Reading EN and JH texts can be often performed with otheractivities, for example, writing and browsing.
As shown in Figure 3, eye blinks (quick up and down eye
movements measured in EOG_V) and forward and backward
saccades to the reading direction have successfully appeared
during a static condition. But dynamic movements of thehead or the glasses cause artifacts, which should be removed
before calculating features.
Figure 4 represents 11-point interpolated recall-precisiongraphs for each activity detection task for each participant.
Although there is not much difference between classification
accuracies of a user-independent and a user-dependant training, the performances are highly distributed by participants.
5CONCLUSION

This paper presented a dataset which contains sensor signals
of JINS MEME electrooculography glasses and corresponding ground truth activity labels. We demonstrated how challenging the natural reading detection is by evaluating ourstatistical feature based approach.
ACKNOWLEDGMENTS
This work is supported by JST CREST and JSPS KAKENHI(Grant Numbers: JPMJCR16E1, 17K12728).

Ishimaru et al.

1.0

1.0

0.9

0.9

0.9

0.8

0.8

0.8

0.7

0.6

Precision

1.0Precision

Precision

UbiComp/ISWC ’19 Adjunct, September 9–13, 2019, London, United Kingdom

0.7

0.6

0.5

0.6

0.5
0.00.2

0.4

0.6

0.8

1.00.50.00.2

Recallp1 (AP: 0.75)
p2 (AP: 0.59)
p3 (AP: 0.72)
p4 (AP: 0.81)

p5 (AP: 0.69)
p6 (AP: 0.70)
p7 (AP: 0.70)
p8 (AP: 0.73)(a) English (EN)

0.7

0.4

0.6

0.8

1.00.00.2

Recallp9 (AP: 0.61)
p10 (AP: 0.70)
Ave. (AP: 0.67)

p1 (AP: 0.83)
p2 (AP: 0.59)
p3 (AP: 0.71)
p4 (AP: 0.79)

p5 (AP: 0.69)
p6 (AP: 0.67)
p7 (AP: 0.70)
p8 (AP: 0.68)

0.4

0.6

0.8

1.0

Recallp9 (AP: 0.58)
p10 (AP: 0.63)
Ave. (AP: 0.65)(b) Japanese horizontal (JH)

p1 (AP: 0.72)
p2 (AP: 0.75)
p3 (AP: 0.78)
p4 (AP: 0.88)

p5 (AP: 0.70)
p6 (AP: 0.83)
p7 (AP: 0.80)
p8 (AP: 0.79)

p9 (AP: 0.62)
p10 (AP: 0.75)
Ave. (AP: 0.71)(c) Japanese vertical (JV)

Figure 4: Recall-precision curves and average precisions of reading detection with user-independent trainingREFERENCES[1] Olivier Augereau, Charles Lima Sanches, Koichi Kise, and Kai Kunze.
2018. Wordometer Systems for Everyday Life. Proceedings of the ACM
on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2018),
123:1–123:21.
[2] Malcolm Brown, Michael Marmor, Eberhard Zrenner, Mitchell Brigell,Michael Bach, et al. 2006. ISCEV standard for clinical electrooculography (EOG) 2006. Documenta Ophthalmologica 113, 3 (2006),
205–212.
[3] Andreas Bulling, Jamie A Ward, Hans Gellersen, and Gerhard Troster.
2011. Eye movement analysis for activity recognition using electrooculography. IEEE Transactions on Pattern Analysis and MachineIntelligence 33, 4 (2011), 741–753.
[4] Anne E Cunningham and Keith E Stanovich. 1998. What reading doesfor the mind. American Educator 22 (1998), 8–17.
[5] Shoya Ishimaru, Kensuke Hoshika, Kai Kunze, Koichi Kise, and Andreas Dengel. 2017. Towards reading trackers in the wild: detecting
reading activities by EOG glasses and deep neural networks. In Proceedings of the 2017 ACM International Joint Conference on Pervasiveand Ubiquitous Computing: Adjunct Publication. ACM, 704–711.
[6] Shoya Ishimaru, Kai Kunze, Koichi Kise, and Andreas Dengel. 2016.
The wordometer 2.0: estimating the number of words you read in reallife using commercial EOG glasses. In Proceedings of the 2016 ACM
International Joint Conference on Pervasive and Ubiquitous Computing:
Adjunct Publication. ACM, 293–296.
[7] Shoya Ishimaru, Kai Kunze, Yuji Uema, Koichi Kise, Masahiko Inami,and Katsuma Tanaka. 2014. Smarter eyewear: using commercial EOG
glasses for activity recognition. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing:
Adjunct Publication. ACM, 239–242.
[8] Shinichiro Kanoh, Susumu Ichi-nohe, Shunsuke Shioya, Kazutaka Inoue, and Ryota Kawashima. 2015. Development of an eyewear tomeasure eye and body movements. In 2015 37th Annual International
Conference of the IEEE Engineering in Medicine and Biology Society.
IEEE, 2267–2270.
[9] Conor Kelton, Zijun Wei, Seoyoung Ahn, Aruna Balasubramanian,Samir R Das, Dimitris Samaras, and Gregory Zelinsky. 2019. Reading
detection in real-time. In Proceedings of the 11th ACM Symposium onEye Tracking Research & Applications. ACM, 43:1–43:5.
[10] Nataliya Kosmyna, Utkarsh Sarawgi, and Pattie Maes. 2018. AttentivU: Evaluating the Feasibility of Biofeedback Glasses to Monitorand Improve Attention. In Proceedings of the 2018 ACM International
Joint Conference and 2018 International Symposium on Pervasive andUbiquitous Computing and Wearable Computers. ACM, 999–1005.
[11] Kai Kunze, Hitoshi Kawaichi, Kazuyo Yoshimura, and Koichi Kise.
2013. The Wordometer–Estimating the Number of Words Read Using
Document Image Retrieval and Mobile Eye Tracking. In Proceedings ofthe 12th International Conference on Document Analysis and Recognition.
25–29.
[12] Oscar D Lara and Miguel A Labrador. 2013. A survey on human activity
recognition using wearable sensors. IEEE Communications Surveys
and Tutorials 15, 3 (2013), 1192–1209.
[13] Juyoung Lee, Hui-Shyong Yeo, Murtaza Dhuliawala, Jedidiah Akano,
Junichi Shimizu, Thad Starner, Aaron Quigley, Woontack Woo, andKai Kunze. 2017. Itchy nose: discreet gesture interaction using EOG
sensors in smart eyewear. In Proceedings of the 2017 ACM International
Symposium on Wearable Computers. ACM, 94–97.
[14] Soha Rostaminia, Alexander Lamson, Subhransu Maji, Tauhidur Rahman, and Deepak Ganesan. 2019. W!NCE: Unobtrusive Sensing ofUpper Facial Action Units with EOG-based Eyewear. Proceedings ofthe ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
3, 1 (2019), 23:1–23:26.
[15] Junichi Shimizu, Thad Starner, Juyoung Lee, Woontack Woo, Murtaza
Dhuliawala, Kai Kunze, and Andreas Bulling. 2016. Solar systemsmooth pursuit interactions using EOG glasses. In Proceedings of the2016 ACM International Joint Conference on Pervasive and UbiquitousComputing. 369–372.
[16] Namrata Srivastava, Joshua Newn, and Eduardo Velloso. 2018. Combining Low and Mid-Level Gaze Features for Desktop Activity Recognition.
Proceedings of the ACM on Interactive, Mobile, Wearable and UbiquitousTechnologies 2, 4 (2018), 189:1–189:27.
[17] Julian Steil and Andreas Bulling. 2015. Discovery of everyday humanactivities from long-term visual behaviour using topic models. In Proceedings of the 2015 ACM International Joint Conference on Pervasiveand Ubiquitous Computing. ACM, 75–85.
[18] Benjamin Tag, Andrew W Vargo, Aman Gupta, George Chernyshov,Kai Kunze, and Tilman Dingler. 2019. Continuous Alertness Assessments: Using EOG Glasses to Unobtrusively Monitor Fatigue Levels
In-The-Wild. In Proceedings of the 2019 CHI Conference on HumanFactors in Computing Systems. ACM, 464:1–464:12.