The Wordometer – Estimating the Number of Words ReadUsing Document Image Retrieval and Mobile Eye Tracking
Kai Kunze, Hitoshi Kawaichi, Kazuyo Yoshimura, Koichi Kise
Osaka Prefecture University, Japankunze,kawaichi,yoshimura@m.cs.osakafu-u.ac.jp
kise@cs.osakafu-u.ac.jp
Abstract—This paper introduces the Wordometer, a novel
method to estimate the words a user reads using the eye gazerecorded by a mobile eye tracker and document image retrieval.
We present a reading detection algorithm which works with over91 % accuracy over 10 test subjects using 10-fold cross validation.
We implement two algorithms to estimate the read words using aline break detector. A simple version gives an average error rateof 13,5 % for 9 users over 10 documents. A more sophisticatedword count algorithm based on support vector regression with an
RBF kernel reaches an average error rate from only 8.2 % (6.5% if one test subject with abnormal behavior is excluded). The
achieved error rates are comparable to pedometers that countour steps in our daily life. This means, the Wordometer can beused as a step counter for the information we read to make ourknowledge life healthier.

I.

I NTRODUCTION

To make their life healthier, more and more people areusing pedometers, be it a dedicated device or a smart phone
application 1 . A survey of medical pedometer studies showsthat monitoring something as simple as how many steps you
take can have a huge impact in health. People wearing pedometers walk in average over 1.5 km more per day, substantiallydecreasing their risk for heart attacks, type2 diabetes and otherdiseases related to obesity [1].
A pedometer is a very simple, yet effective tool to monitor
and improve our physical fitness. We want to provide a similar
tool for our cognitive fitness: the Wordometer, counting how
many words a user reads per document per day. This paper
presents our initial work towards this goal.
Of course, how much we read directly influences the sizeof our vocabulary and our language skills [2]. However, several
studies also indicate that the more people read throughout theday the higher are their general knowledge and critical thinking
skills [2], [3]. Interestingly, similar effects could so far not
been shown for TV or other video/multimedia consumption(even when focusing on documentaries). Being able to justcount the words we read each day would help to assess thegeneral knowledge of a person, as there are strong correlations
between the two [3].
Exploring the reading activities of people, in general, anda Wordometer in particular is also an interesting for the document analysis, as we would be able to provide more insides
into how/when documents are read (e.g. number of words readper page statistics or “this is the most read paragraph/page inthis book”).
1 http://fitbit.com

The main contributions we present in this paper are asfollows:1)
2)
3)
4)

We present an algorithm based on eye gazes obtainedby a mobile eye tracker to segment reading from notreading.
We present a method to segment read lines using theuser’s eye gaze and document image retrieval.
Based on the line estimates, we developed 2 methods
to estimate the number of words a user reads usinga mobile eye tracker and document image retrieval.
Our best algorithm to estimate word numbers has anerror rate of only between 6 -8 % evaluated in a study
with 10 users and 10 documents.

Achieving an error rate of 6-8 % for our Wordometer isreasonable comparing it with the pedometer error rate which
is between 3-10 % [1]. The pedometer studies are by far more
representative. Yet, this is the first time to our knowledge
somebody tired to estimate the amount of words read and6-8% seem to be a good error margin for a problem moredifficult then counting steps. Section II is devoted to describe
our approach to implement the Wordometer. Sections III andIV are for experimental results and discussions. Section V isto sum up what we have learnt from this research activity.
II.

A PPROACH

We are interested in the number of words read. We could
use optical character recognition/ text detection or similar
technologies to record all characters a user sees assumingthat the amount of text surrounding people is indicative forhow many words they read. However, this will only be arough estimate at best. It is easy to think of scenarios where
this utterly fails. Imagine an American tourist in a Japanesecity with a lot of billboards, although he’s exposed to a lotof text, the words read are close to zero (assuming he does
not understand Japanese). Therefore, we have to monitor thereading –the decoding of letters, words and assigning meaning
to them.
As reading is a cognitive process, sensing brain activity
seems to provide the most insight. However, directly sensing
brain activity can only be done by more or less invasive
methods (e.g. electroencephalography, functional magnetic resonance imaging, electrocorticography). On the other hands,
eye movements and gaze are strongly correlated with reading
and text comprehension [4], [5], [6] and it can be done byrelatively unobtrusive already today.
Our approach has several discrete steps:

TABLE I.

F EATURES FOR R EADING D ETECTION

fixation related features

saccade related features

waveletfeaturethe number of fixations
sum of the duration of fixations
average time of fixations
average length of saccades
minimum length of saccades
horizontal element of saccades
vertical element of saccades
average amplitude of wavelets

eyetracker

Fig. 1.

Eye fixations and saccade traces while reading a document.

User’s&eyes
Fig. 3.

Fig. 2.

1)
2)
3)

4)
5)

Eyetracking data for Reading (left) versus not Reading (right)

The raw eye gaze data is summarized into fixations
fixations and saccades according to Busher et al. [7](see Figure 1).
On basis of the optioned eye gaze data we calculatefeatures and classify reading from not reading.
On reading data, we use the video recording fromthe eye tracker to apply a document image retrieval
technique (this is used to filter some head movements
and get the average word count by line for thedocument).
A line detection algorithm is applied on the rectified
eye gaze data.
Using the numbers of lines detected we calculate
the words per line using 2 methods: the first bymultiplying it with the average words per line ofthe document, the second by using support vector
regression.

A. Reading Detection
There are already a couple of methods in the related
work [8], [9]. However, they work with different sensing
modalities (e.g. electrooculography) and eye tracking hardware. Therefore we decided to implement our own algorithm.
The process is straight forward: we calculate eye gazefeatures given in Tabel I over a 3sec. frame sliding window
and apply a Support Vector Machine classifier with a radialbasis function on the resulting feature vector.
B. Applying document image retrieval
With the help of an eyetracker and document image retrieval, we are able to obtain information on the document theuser is reading as well as his/her eye gaze information.

gaze

Conversion of an eye gaze by using LLAH.

We use a document image retrieval method based on“Locally Likely Arrangement Hashing” (LLAH) to associate
the paper with the digital document [10]. We use the video
feed from the eye tracker as input to LLAH. LLAH retrieves
the corresponding page from a document image database bycomparing feature points. The eye gaze data obtained by theeye tracker is represented in the coordinate system of a scene
camera of the eyetracker. We use LLAH to convert the eye gazecoordinates to the corresponding coordinates of the document
by using a homography estimate.
Figure 3 illustrates an example of an eye gaze converted
from a camera captured image (left) to its corresponding
retrieved document image (right). On the left, the rectangle represents the focused area in which the corresponding document
is retrieved. With this method we avoid potential confusioncaused by other documents also captured in the image. By
using this capability, we can record eye gazes on the coordinate
system of a retrieved document (see Figure 4 for an example).

C. Detection of Line Breaks
Figure 5 shows a typical eye movement when a person
reads over line breaks. In our proposed method, we segmenta sequence of eye movements into parts corresponding to textlines by analyzing them on the coordinate system of a retrieved
document.
The eye movement for a line break is against the regularreading order from the end of the text line backwards to thestart of the new line. We detect such a movement using thefollowing line break detection.
Let (g1 , ..., gl ) represent a subsequence of gaze points. Acurrent gaze gl is recognized as a line-break if the following
conditions are satisfied:1)

the area of a rectangle that circumscribes the l gazes
g1 , ..., gl is larger than a certain threshold,TABLE II.

gaze information

F EATURES

duration required for reading
the number of fixations for a page
total distance of eye movements
total distance of saccades
average distance of saccades

Fig. 4. Two examples of an eye gaze conversion by using LLAH. The rightsample works very well, in the left sample an offset is introduced.

Fig. 6.

Line%break%
Fig. 5.

2)

Example of eye gaze data from a user reading over a line breakthe direction of eye movements from the l − k th tothe l th gaze is opposite. Some succeeding gazes arealso included as a part of a line break.

D. Wordometer
Next, we apply our line break detector to build the Wordometer.
In general, there are many ways to implement a Wordometer. If we are able to record all read words accurately, it istrivial to realize the Wordometer based on the record. However,
it is not easy to achieve such an accuracy even with the help ofdocument image retrieval. If we are interested in just counting
the rough number of read words, it is not necessary to havehigh accuracy; some errors are acceptable just as is the caseof pedometers.
In this research we implemented two methods for a Wordometer based on the line break detector as follows.
First we build a very simple method. Let N be the average
number of words in one line of a page the user is reading, andL be the number of estimated line breaks. Note that N can beobtained by using document image retrieval. The number ofread words is estimated as N (L+1). The more errors we havein the number of estimated line breaks, the estimated numberof read words gets more inaccurate.
The second method is more sophisticated. In order toimprove the accuracy, we employ a feature vector x that
represents eye movement information and estimate the numberof read words by a function f (N, L, x), which is learnt byusing the learning samples. As a learner, we employ the SVR(support vector regression). In the following this method iscalled the SVR-based method.
Table II shows features in the vector x we obtain from eyemovement data.
III.

E XPERIMENTS

We asked 10 subjects to read 10 pages of documents
written in English and record their eye movements by using

the SMI iViewX Eye Tracking Glasses used for the experiments

the SMI iViewX ETG (Eye Tracking Glasses), see Figure 6.
Among the subjects, 7 are with unaided vision, 2 are withcontact lenses on both eyes, and one with a contact lens on asingle eye. Documents are from tests of PET (Preliminary English Test) in ESOL(English for Speakers of Other Languages)
by University of Cambridge. For each document we have atime limit defined based on the number of words included init.
For the reading detection we designed another experiment
including also the 10 subjects reading 10 documents from ajob hunting test. Other than that the conditions are the same.
The not reading activity included looking around in the room,
playing with objects (e.g. cellphone etc.).
The following is the procedure of experiments. Each subject was requested to wear the eyetracker for calibration. Thenhe/she is requested to read a document placed on a desk infront of him/her and with a natural posture. The calibration ofthe eyetracker is applied after reading each document. Then
we applied k-fold cross validation with k subjects.
IV.

R ESULTS AND D ISCUSSION

A. Reading Detection
Reading versus not reading can be easily distinguished justby visualizing the the eye gaze data for the different classes, asdepicted in Figure 2. As expected the reading detection works
reliably well with an average of 91 % over the 10 subjects,
see Table III for details.
B. Line break estimation
One test subject with unaided vision often fell asleep during
the data recording and his recorded data showed a lot ofinconsistencies. Therefore, we decided to exclude him and thedata from the remaining 9 subjects are utilized for 9-fold crossvalidation.
Table IV shows errors observed during the line breakdetection. The average errors of the estimated number of lines
is 2.1 lines per document. Since the average number of linesof documents is 17, we are able to estimate the number oflines with 12.4% errors.
From the Table IV , we can observe that the results by thesubject h are much worse than other results. His data includes

TABLE IV.
doc.
# lines
A15B15C
12D19E
18F19G
18H
17I
16J
21Ave. errors

TABLE III.
User
Accuracy(%)

1
100

2100

a-1-1
3-3-1-62-1
1-1-0.8

b2
2
10
7
1-28
5254

E RRORS OF LINE BREAK DETECTION .
c001-21-3-1
102-0.1

d123-21102
551.8

R EADING D ETECTION R ESULTS
3100

480

5100

680

775

8100

9100

1075

All91

more frequent eye blinks than any other subjects. In addition,
he often re-read the end of the line just after the line break.
This behavior erroneously increased the number of detectedline breaks.
Another source of errors are short lines. The proposed
method tends not to detect such short lines so that the resultant
estimated number is smaller than the groundtruth. The reason
is two-fold:1)
2)

The detection is based on the area of a bounding boxof gazes. Since short lines only have small numberof gazes and thus the area is sometimes too small.
For most subjects, the duration of gazing a short line
is too small.

The results for document C are clearly worse compared toother documents. Subjects b and h often went back to previouslines and read them for this document again.
C. Wordometer
Table V shows the number of read words for each document estimated by the simple method. The average error is
47.95 words. Since the average number of words in a document
is 279.3, the error is 17.2%. Table VI shows the number ofread words for each document estimated by the SVR method.
The average error is 40.5 words (14.5%) thus the SVR method
is superior.
Table VII shows the number of estimated words for all
documents. For the SVR method we achieve an average errorof 8.2 %, however we get a really high error rate from subject
h due to him also re-reading several lines. If we remove subject
h from the analysis we are can improve the error to just about
6.5 %.
V.

subject
ef3-1
2202
10-1
1-1-2212224201.2
0.9

R ELATED W ORK

A good overview about controlled studies in psychology
about eye movements in relation to reading is given by
Rayner [11] Kligel et al. explore the relations of eye fixations
and mental activities during reading [12].
The existing work so far focuses on controlled lab experiments on specific population groups –mostly elderly andg01
10-1-3-1000-0.3h10
11
1351010257108.3

i82
64423
5
1
74.2

Ave. error
for each doc.
2.4
2.3
4.3
1.1
1.6
-0.4
1.7
2.3
2.4
3.3
2.1

children– trying to diagnose specific disease or mental states(attention deficit, alzheimer etc.) [5], [4]. There is some work
showing the strong relation between how much a person
reads and their language skills, general knowledge and criticalthinking skills [11], [2], [4].
The closest to the work presented here is by Bulling et.
al. They present a method to for reading detection using
electrooculography [13]. Although their work looks at reading
detection in a realistic environment, they do not try to estimate
the word count.
Other research classifies the different types of reading,skimming etc. [14], [8]. Xu et al. use eye tracking in a controlled lab environment to create summaries of documents [15].
VI.

C ONCLUSION AND F UTURE W ORK

We presented a reading segmentation algorithm, a line
break detector and two methods to implement a Wordometer.
In conclusion, our error margin for the Wordometer is 8.2% ( or 6.5 % removing one test subject). Comparing it to
the effectiveness of a pedometers in realistic scenarios ( error
rate of 3-10 % ), we believe 8.2 % is a good baseline forcounting words regarding the impact a Wordometer could haveon quantifying language skills and general knowledge.
For future work we want to overcome the limitations ofdocument image retrieval (all documents the user reads need
to be registered with the system). Currently our Wordometer
methods still rely on document image retrieval for:1)2)

the correction of the eye gaze (translation to thedocument coordinate system)
the average word count per document.

For the eye gaze correction we can use optical flow. There isrelated work that shows the successful application of opticalflow to filter out head movements during tomography [?]. The
simple solution regarding the average word count per line isto apply a constant. Yet this will likely increase the error rate.
We will also try an estimation of words per line using the eye
fixations and saccades per line.
ACKNOWLEDGMENT
This work is supported in part by the CREST project fromJapan Society for the Promotion of Science (JSPS).

TABLE V.

N UMBER OF READ WORDS FOR EACH DOCUMENT ESTIMATED BY THE SIMPLE METHOD .

doc.
# words
A
255
B
203C
210
D302
E
296
F
276
G
281
H
298
I
255
J417
ave. error

TABLE VI.

a
238.0
189.4
262.5
219.6
279.5
188.8
312.2
280.5
270.9
397.1
35.4

b
289.0
230.1
385.0
356.9
312.4
246.9
405.9
385.6
286.9
516.3
68.0

c
255.0
203.0
227.5
247.1
312.4
246.9
265.4
315.5
255.0
456.7
19.1

subject
e
306.0
230.1
210.0
274.5
279.6
261.5
312.2
333.1
286.9
456.7
27.4

d334.7230.1
262.5
233.4
312.4
290.5
281.0
333.1
272.0
516.3
41.0

g
255.0
216.5
227.5
260.8
279.6
232.4
265.4
298.0
255.0
417.0
14.8

h
366.6
351.9
437.5
329.5
460.4
421.3
312.2
385.6
425.0
615.6
131.3

i
391.0
230.1
315.0
315.7
361.8
305.1
327.8
385.6
270.9
556.0
66.6

doc-wiseave. error
32.9
49.1
61.8
40.5
33.8
28.9
23.2
29.3
21.3
70.6
47.95

N UMBER OF READ WORDS FOR EACH DOCUMENT ESTIMATED BY THE SVR- BASED METHOD .

doc.
# words
A
255B
203C
210
D
302
E
296
F
276G
281
H
298
I
255
J
417
ave error

a196.7
205.9
249.0
153.2
237.3
216.5
231.1
250.7
230.1
249.6
65.6

TABLE VII.

simple method
errors by the simple method
SVR-based method
errors by the SVR-based method

a2638.8
154.2(5.5%)
2220.4572.6(20.1%)

b
266.0
227.6
309.9
289.9
283.3
235.5
310.5
318.3
269.3
376.8
30.5

c
278.9
277.0
277.5
277.9
281.3
279.1
279.0
281.3
278.8
285.2
38.1

subject
e
255.6
252.8
252.8
252.8
252.8
252.8
252.8
252.8
252.8
252.8
44.8

d
288.9
257.2
232.4
281.1
266.9
266.4
250.9
271.8
249.5
391.6
25.7

f
276.0
236.4
253.1
249.1
276.0
242.8
265.6
286.6
249.1
338.2
31.5

g
241.8
260.9
221.1
275.2
266.6
264.4
256.9
272.7
244.4
373.5
25.4

h
316.2
310.6
356.2
298.8
370.8
353.8
275.4
336.4
341.6
441.8
62.6

i
327.8
240.4
294.2
275.5
317.9
278.1
305.2
330.7
272.6
406.1
33.0

doc-wiseave. error
54.834.6
75.9
45.8
38.4
46.8
34.7
44.8
33.3
76.3
40.53

E STIMATED NUMBER OF READ WORDS FOR ALL DOCUMENTS
b3415.1
622.1(22.2%)
2887.594.5(3.4%)

c2784.6
8.4(0.3%)
2796.03.0(0.1%)

d3065.9
272.9(9.7%)
2756.836.2(1.3%)

R EFERENCES[1]

f318.8
230.1
245.0
260.8
312.4
246.9
296.6
333.1
238.0
417.0
28.0

D. M. Bravata, C. Smith-Spangler, V. Sundaram, A. L. Gienger, N. Lin,R. Lewis, C. D. Stave, I. Olkin, and J. R. Sirard, “Using pedometers
to increase physical activity and improve health,” JAMA: the journalof the American Medical Association, vol. 298, no. 19, pp. 2296–2304,2007.

[2]

A. Cunningham and K. Stanovich, “What reading does for the mind,”
Journal of Direct Instruction, vol. 1, no. 2, pp. 137–149, 2001.

[3]

P. Terenzini, L. Springer, E. Pascarella, and A. Nora, “Influences
affecting the development of students’ critical thinking skills,” Researchin higher education, vol. 36, no. 1, pp. 23–39, 1995.

[4]

A. R. Clarke, R. J. Barry, R. McCarthy, and M. Selikowitz, “Eeganalysis of children with attention-deficit/hyperactivity disorder andcomorbid reading disabilities,” Journal of Learning Disabilities, pp.
276–285, 2002.

[5]

E. C. Ferstl, J. Neumann, C. Bogler, and D. Y. von Cramon, “Theextended language network: A meta-analysis of neuroimaging studies
on text comprehension,” Human Brain Mapping, pp. 581–593, 2008.

[6]

O. Dimigen, W. Sommer, A. Hohlfeld, A. Jacobs, and R. Kliegl,“Coregistration of eye movements and eeg in natural reading: analyses
and review.” Journal of Experimental Psychology: General, vol. 140,
no. 4, p. 552, 2011.

[7]

G. Buscher and A. Dengel, “Gaze-based filtering of relevant document
segments,” in Workshop on Web Search Result Summarization andPresentation. Workshop on Web Search Result Summarization andPresentation (WSSP-2009), located at in conjunction with WWW, vol. 9,
2009, pp. 20–24.

[8]

R. Biedert, J. Hees, A. Dengel, and G. Buscher, “A robust realtime
reading-skimming classifier,” in Proc. of ETRA ’12, 2012, pp. 123–130.

[9]A. Bulling, J. Ward, H. Gellersen, and G. Troster, “Eye movement
analysis for activity recognition using electrooculography,” Pattern

[10]

[11]
[12]

[13]

[14]
[15]

subjecte2950.5
157.5(5.6%)
2530.8262.2(9.4%)

f
2898.7105.7(3.8%)
2672.9120.1(4.3%)

g2707.285.8(3.1%)
2677.4115.6(4.1%)

h4105.51312.5(47.0%)
3401.5608.5(21.8%)

i
3459.0666.0(23.8%)
3048.5255.5(9.1%)

ave. error
376.1(13.5%)
229.8(8.2%)

Analysis and Machine Intelligence, IEEE Transactions on, vol. 33, no. 4,pp. 741–753, 2011.
T. Nakai, K. Kise, and M. Iwamura, “Use of affine invariants in locallylikely arrangement hashing for camera-based document image retrieval,”
In Proc. of DAS 2006, vol. 3872, pp. 541–552, Feb. 2006.
K. Rayner, “Eye movements in reading and information processing: 20years of research.” Psychological bulletin, vol. 124, no. 3, p. 372, 1998.
R. Kliegl, A. Nuthmann, and R. Engbert, “Tracking the mind during
reading: the influence of past, present, and future words on fixation
durations.” Journal of Experimental Psychology: General; Journal ofExperimental Psychology: General, vol. 135, no. 1, p. 12, 2006.
A. Bulling, J. A. Ward, H. Gellersen, and G. Tröster, “Robust recognition of reading activity in transit using wearable electrooculography,”in Proc. of Pervasive ’08, 2008, pp. 19–37.
G. B. Duggan and S. J. Payne, “Skim reading by satisficing: evidence
from eye tracking,” in Proc. of CHI 2011, 2011, pp. 1141–1150.
S. Xu, H. Jiang, and F. C. Lau, “User-oriented document summarization
through vision-based eye-tracking,” in Proc of IUI, ser. IUI ’09, 2009,
pp. 7–16.