Work-in-Progress

CHI 2015, Crossings, Seoul, Korea

Towards Extraction of Subjective
Reading Incomprehension:
Analysis of Eye Gaze Features
Ayano Okoso
Osaka Prefecture Univ.
1-1 Gakuen-cho, Naka, Sakai
Osaka, Japanokoso@m.cs.osakafu-u.ac.jp

Joachim Folz
DFKI GmbH
Trippstadter Strasse 122
Kaiserslautern, Germany
joachim.folz@dfki.de

Takumi Toyama
DFKI GmbH
Trippstadter Strasse 122
Kaiserslautern, Germany
takumi.toyama@dfki.deMarcus Liwicki
DFKI GmbH
Trippstadter Strasse 122
Kaiserslautern, Germany
marcus.liwicki@dfki.de

Kai Kunze
Keio Media Design
Keio University
Yokohama, Japan
kai.kunze@gmail.com

Koichi Kise
Osaka Prefecture Univ.
1-1 Gakuen-cho, Naka, Sakai
Osaka, Japan
kise@cs.osakafu-u.ac.jp

Abstract
One way to optimize learning processes is to clearly informthe learner about problematic areas. Recent work on
gaze-based CHI showed that a reader’s language skill canbe inferred by gaze analysis. However, only few
approaches have been proposed to identify those
document parts a reader finds problematic. Our goal is todevelop a computational method for reading
incomprehension extraction. As initial work, we analyze
which eye gaze features are useful for such part-based
reading incomprehension extraction at three levels of
document structure: paragraphs, segments and words.

Author Keywords
Eye tracking, subjective reading comprehension, gazefeatures, reading analysis

ACM Classification Keywords
H.5.2 [User Interfaces]: Input devices and strategies.
Permission to make digital or hard copies of part or all of this work forpersonal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. Copyrights for third-party
components of this work must be honored. For all other uses, contact the
Owner/Author. Copyright is held by the owner/author(s).
CHI 2015 , April 18-23, 2015, Seoul, Republic of Korea.
c 2015 ACM 978-1-4503-3146-3/15/04 ...$15.00.
Copyright ⃝http://dx.doi.org/10.1145/2702613.2732896

Introduction
The eyes play a central role for information processing inreading: humans eﬀectively control them to collect data
to gradually build an understanding of the text.
Considerable eﬀort has been made to reveal the nature ofhuman eye movements while reading text [3, 4, 9]. Insmooth reading, it is said that eye fixations last about1325

Work-in-Progress

In classical eye movement research, there are two classes oftypical eye movements: fixations and saccades. Saccades
are rapid eye movements that
occur when a viewer switches
the target of focus. Fixations
are the states that the eyes remain relatively still and occur
between saccades.

CHI 2015, Crossings, Seoul, Korea

200-250 ms and the mean saccade size is 7-9 letter
spaces [9]. When the reader finds it diﬃcult to understandthe text, such smooth eye movements may be disrupted.
Based on prior studies for eye movements during reading,research revealed the power of eye gaze-based approaches
for assessing reading comprehension, language skill, andtext quality. Biedert et al. presented a method formeasuring objective quality of written text [1]. They
aggregated gaze features during reading of an entire text
area to find out which text passages are comprehensible
and which are not. Although the exact degree ofcomprehension of text is in part subjective, they showedthat gaze-based objective measurement is also reliable.
Martinez-Gomez and Aizawa showed the potential of eye
gaze analysis for subjective understanding level
recognition [6]. Their work mainly focused on recognitionof a reader’s language level, where the comprehension ofindividual text parts is dismissed. The authors also
discussed linguistic features compared to eye gaze ones.
However, those linguistic features were not founddiscriminative in their experiments. Similarly, Kunze et al.
presented a method for inferring language expertise fromreading behavior [5]. Furthermore, they proposed an
approach for spotting diﬃcult words for the reader.
Inspired by the above mentioned prior work, we prototype
an eye gaze-based reading comprehension assessment
approach. Particularly, we aim to extract subjective
part-based reading incomprehension (i.e., detecting partsof a document that are not understood by the reader) on
multiple document structure levels: does the reader havediﬃculty understanding a paragraph, sentence or word. In
this paper, we analyze eye gaze features as a preliminary
for formulation of such an extraction method.

Figure 1: Top: data recording
setting. Bottom: a sample imageof the document we used in theanalysis.

We record eye gaze data of participants reading English
documents. Afterwards they also answer test questions
regarding the content. Then we ask the participants tomark which paragraphs, sentences/clauses, and words
they find diﬃcult. Finally we extract eye gaze featuresfrom the recording and analyze them with respect todiﬃculty as annotated by the reader.

Data Collection
Our data collection procedure is as follows: As shown in
Figure 1, we set a stationary eye tracker (SMI RED 250)
and a display on the desk.First, the eye tracker iscalibrated for the participant. Since our eye tracker does
not have a head motion compensation function, we ask
the participant to keep his/her head as still as
possible.After checking the calibration, we start a
recording session. For each recording, the participant
reads a single page document presented in full-screen
mode. Subsequently, he/she answers several questions
regarding the content. In total, the participant completes
10 English language documents. Documents andquestions were sourced from the Test of English forInternational Communication (TOEIC). A sample imageof a document page is shown in Figure 1.
Every time a recording is completed (document is readand questions are answered), we also ask the participant
to mark diﬃcult parts in the document. We asked forthree types of annotations for diﬀerent document
structure levels: 1) diﬃcult (e.g. unknown) words, 2)diﬃcult sentences or clauses, and 3) the most diﬃcult
paragraph in the document. Note that 3) was chosen
instead of labeling individual paragraphs, because someparticipants had problems to tell whether a specificparagraph was diﬃcult or not. A sample document
annotated like this is shown in Figure 2.

1326

Work-in-Progress

CHI 2015, Crossings, Seoul, Korea

Table 1: Extracted eye gaze features at three document structure levels.

Level

FeatureDescription

Fixation

fixation duration (ff d )pupil size (fps )fixation duration (microsec.)
diameter of the pupils (pixel)

Segment

number of fixations (snf )avg. fixation duration (sf d )avg. saccade length (ssl )number of fixations in a segment
average of fixation duration in a segment (microsec.)
average of saccade length in a segment (pixel)

Paragraphnumber of fixations (pnf )avg. of fixation duration (pf d )avg. saccade length (psl )total reading duration (ptd )avg. number of fixations per word (pf w )avg. reading duration per word (pdw )number of regressions (pnr )total number of words (pnw )number of fixations in a paragraphaverage of fixation duration in a paragraph (microsec.)
average of saccade length in a paragraph (pixel)
total fixation duration (microsec.)——number of backward saccades (excluding line break)total number of words in a paragraphWe recorded gaze data from seven participants. However,
we only focus one representative reader in this preliminary
analysis as work-in-progress.

First, we detect fixations and saccades from the recorded
gaze data using a dispersion approach [2]. Then,
individual features are calculated based on detected
fixations and saccades.

Feature Extraction
Figure 2: Annotations of diﬃcultparts in a document. Diﬃcultwords are circled. Diﬃcultsentences (clauses) are
underlined. The most diﬃcult
paragraph is marked by a circle
next to it.

We use the following three gaze structure levels torepresent the diﬀerent document parts: fixations (i.e.
words), gaze segments (i.e. lists of words, clauses,
sentences), and paragraphs. Gaze segments are used since
they are similar to sentences in terms of document
structure level, but allow more flexibility in our approach.
For each level, we extract gaze features and analyze the
distributions of individual feature values with respect todiﬃcult/non-diﬃcult document parts. Table 1 shows asummary of the features we extract in this analysis.

Fixation Level Gaze Feature
The smallest unit of gaze feature in this work is a fixation.
During reading, a reader continuously fixates on a word toprocess information. When a problem for word
comprehension occurs, a fixation duration may belonger [3]. We also extract pupil size during fixation. We
suspect that it corresponds to the reader’s stress level, so
it may change if he/she is confronted with an unknown
word.

1327

Work-in-Progress

CHI 2015, Crossings, Seoul, Korea

Segment Level Gaze Feature
On this level, we want to extract reading incomprehension
for an arbitrary document part. Thus, instead of
considering “sentence” or “clause” as our unit, we extract
features based on a gaze segment. A gaze segment is asequence of consecutive fixations within Wt seconds.
Based on a preliminary test, we set Wt = 7.5 in thisanalysis.

Figure 3: Detected fixations(circles) are not always locatedexactly on a word. However, we
can see which paragraph the
fixations belong to.

Paragraph Level Gaze Feature
Recorded raw gaze data can be noisy, because of e.g. theinaccuracy of the eye tracker or the natural eye gazebehavior of the participant. For instance the gaze is notnecessarily fixated on the center of a word region [7].
Thus, it can be hard to tell which word a fixation belongsto (see also Figure 3). Previous work showed that gazedata can be robustly associated with a paragraph, since
spaces between paragraphs are relatively large in a normal
document [8]. Therefore, we consider that aparagraph-level is the smallest unit which can containdocument features. The total number of words pnw is oneof such document feature.

Analysis

Figure 4: Fixation labeling.
Fixations at time t (top) andt + 1 (bottom). Although the
new fixation at t + 1 is closer to
February than seasons, priorfixations suggest it should be
associated with seasons by thehuman labeler.

Gaze Data Correction
As previously discussed, gaze samples are too noisy to findthe exact word the reader fixates on. However, for ouranalysis, we need to know which gaze samples areassociated with which words, segments, or paragraphs as
ground truth. If we have such associations between gazeand text, we can analyze the extracted features withregard to the annotations from the participants. We
create such association data by manually labelingindividual fixations. Using a custom labeling tool, ahuman labeler monitors each fixation (and its immediate
predecessors) and assigns the most probable word thereader is fixating as shown in Figure 4. Although we could
also use a filter to automate this manual association
process, the recorded data is too noisy to assure thequality of the resulting ground truth.
Consequently, we create fixation data in which each ofthem is associated with one word. If a fixation isassociated with a diﬃcult word, we consider that the
fixation is on the diﬃcult word. However, note that justbecause it is associated with a diﬃcult word (or segment)does not necessarily mean that the extracted feature
values diﬀer from those associated with non-diﬃcultwords [10]. A reader often ignores or skips a diﬃcult(unknown) word if he/she can understand its context. We
discuss this point in the analysis.
If the ith fixation Fi is associated with a diﬃcult word, welabel that Fi = +1 and otherwise Fi = −1. Similarly, if agaze segment Si contains a diﬃcult word, clause, orsentence, Si = +1. The most diﬃcult paragraph is alsolabeled as Pi = +1.
Analysis of Variance
For each feature, we calculated the average value and thestandard deviation (SD) on both diﬃcult and non-diﬃcultparts. Figure 5 shows all averages and SDs from one
representative participant. For visualization, we normalize
the values using the maximum of each feature. In thisfigure, we also show p-values of individual one-way
ANOVA tests. If the p-value is small (normally,P < 0.05), we could infer that the samples are drawn
from diﬀerent distributions, which means that the value islikely be diﬀerent when the reader has a comprehension
problem. For example, the p-value for the pupil size in a
fixation-level fps is 0.71. Thus, we infer that for thisfeature there is no significant diﬀerence between diﬃcultand non-diﬃcult. On the other hand, for the fixation

1328

Work-in-Progress

CHI 2015, Crossings, Seoul, Korea

Figure 5: Average and standard deviation (SD) of each feature on both diﬃcult (+)
and non-diﬃcult (-) document parts. Error bars represent for SDs. P-values for eachfeature are calculated by ANOVA.

duration ff d , P < 0.01. It statistically shows that there isa significant diﬀerence. The p-value for the average
fixation duration of gaze segments is sf d = 0.47.
Opposite to the fixation-level, there is no significant
diﬀerence in the average of fixation duration.
On the other hand, the number of fixations snf and theaverage saccade length ssl may likely be diﬀerent between
diﬃcult and non-diﬃcult gaze segments. Not surprisingly,
the number of words in a paragraph pnw has a low p-value(P < 0.01). This result shows that a reader is likely tofind a paragraph diﬃcult if it is very long. The total
reading duration of a paragraph ptd also has a low
p-value, likely because a longer paragraph usually takes
longer to read.
However, a short paragraph is sometimes marked as themost diﬃcult by the reader when it has to be re-read.
The number of fixations per word pf w , average fixation

Figure 6: Histogram of ratio of occurrences for
fixation duration in ff d . Orange is non-diﬃcult
one and blue is diﬃcult one.

duration pf d and average saccade length psl have highp-values. As such, features of individual fixations appear
to be less important to assess the diﬃculty on aparagraph level.
Histogram Analysis
As previously mentioned, a reader sometimes ignoreswords that he/she finds diﬃcult. Figure 6 shows thehistogram distribution of fixations associated with
diﬀerent durations ff d . We can see that longer fixations(longer than 0.7 sec.) are associated with diﬃcult words,
whereas the diﬃculty is not clear with medium-length
durations (about 0.4 - 0.7). Therefore, we should consider
that the subjective part diﬃculty is not always inferablefrom eye gaze only when we evaluate the incomprehension
extraction system. It could happen that the reader has
comprehension problems even though eye movements arealmost normal.

1329

Work-in-Progress

Initial Classification Test
In addition to the analysis, we
attempted to recognize diﬃcultparts using a Random Forest(RF) classifier trained on theextracted features as an initialtest. However in this initialtest, the classification was not
successful. We believe that thiswas caused by insuﬃcient or
unfit training data. For future
work in this direction we see
two options: collect more dataor develop an unsupervised approach, which does not require
training data.

CHI 2015, Crossings, Seoul, Korea

Conclusion and Outlook
From the analysis, we found several features that can beeﬀective for extracting reading incomprehension. On low
level document structures (words or segments), featuresof fixations (duration and number of fixations) or
saccades (avg. of saccade length) showed significant
diﬀerences between diﬃcult and non-diﬃcult ones. On
the other hand, on high level structures (paragraphs),
comprehensive features such as total reading duration or
number of words in a paragraph are most promising, while
the previously mentioned values are less discriminative.
Based on this analysis, we are able to select eﬀective
features for each level of document structure. Using our
findings, we will develop a method for subjective reading
incomprehension extraction using the features.

Acknowledgements
This work is partly supported by Kakenhi Grant-in-Aid B
No. 26730095 and the CREST project ”Development ofFundamental Technologies for Innovative Use ofCharacter/Document Media”.

References[1] Ralf Biedert, Andreas Dengel, Mostafa Elshamy, andGeorg Buscher. Towards robust gaze-based objective
quality measures for text. Proceedings of the
Symposium on Eye Tracking Research andApplications - ETRA ’12, p. 201, 2012.
[2] Georg Buscher, Andreas Dengel, and Ludger van
Elst. Eye movements as implicit relevance feedback.
Proceeding of CHI extended abstracts on Human
factors in computing systems - CHI ’08, p. 2991,
2008.
[3] Charles Clifton, Adrian Staub, and Keith Rayner. Eye
movements in reading words and sentences. In Eye
Movements, pp. 341–371. 2007.

[4] M A Just and P A Carpenter. A theory of reading:from eye fixations to comprehension. Psychologicalreview, Vol. 87, pp. 329–354, 1980.
[5] Kai Kunze, Hitoshi Kawaichi, Kazuyo Yoshimura,and Koichi Kise. Towards inferring language
expertise using eye tracking. CHI ’13 Extended
Abstracts on Human Factors in Computing Systems
on - CHI EA ’13, p. 217, 2013.
[6] Pascual Martı́nez-Gómez and Akiko Aizawa.
Recognition of understanding level and language skill
using measurements of reading behavior. In
Proceedings of the 19th international conference onIntelligent User Interfaces - IUI ’14, pp. 95–104, New
York, New York, USA, 2014. ACM Press.
[7] G.W. McConkie, P.W. Kerr, M.D. Reddix, andD. Zola. Eye movement control during reading: I.
The location of initial eye fixations on words. Vision
Research, Vol. 28, No. 10, pp. 1107–1118, January
1988.
[8] Ayano Okoso, Kai Kunze, and Koichi Kise. Implicit
gaze based annotations to support second language
learning. Proceedings of the 2014 ACM International
Joint Conference on Pervasive and Ubiquitous
Computing Adjunct Publication - UbiComp ’14
Adjunct, pp. 143–146, 2014.
[9] Keith Rayner. Eye movements in reading andinformation processing: 20 years of research.
Psychological bulletin, Vol. 124, No. 3, pp. 372–422,
November 1998.
[10] W Schroyens, F Vitu, M Brysbaert, andG D’Ydewalle. Eye movement control during reading:foveal load and parafoveal processing. The Quarterly
journal of experimental psychology. A, Humanexperimental psychology, Vol. 52, pp. 1021–1046,
1999.

1330