Implicit Gaze based Annotations toSupport Second Language
Learning
Ayano Okoso

Abstract

Graduate School of Engineering
Osaka Prefecture University
1-1 Gakuen-cho, Naka, Sakai,Osaka, Japanokoso@m.cs.osakafu-u.ac.jp

This paper explores if implicit gaze based annotations can
support reading comprehension tasks of second language
learners. We show how to use eye tracking to add implicit
annotations to the text the user reads and we start byannotating physical features (reading speed, re-reading,number of fixation areas) to documents using eyetracking.

Kai KunzeGraduate School of Engineering
Osaka Prefecture University
1-1 Gakuen-cho, Naka, Sakai,Osaka, Japanfirstname.lastname@gmail.com

Koichi Kise
Graduate School of Engineering
Osaka Prefecture University
1-1 Gakuen-cho, Naka, Sakai,Osaka, Japan
kise@cs.osakafu-u.ac.jp

We show initial results of an ongoing experiment. So far,
we recorded the eye gaze of 2 students for 2 documents.
We gather initial feedback by presenting the annotated
documents to two English teachers.
Overall, we believe implicit annotations can be a usefulfeedback mechanism for second language learners.

Author Keywords
language expertise, mobile eye tracker, eye movements

Permission to make digital or hard copies of part or all of this work for personalor classroom use is granted without fee provided that copies are not made or
distributed for profit or commercial advantage and that copies bear this noticeand the full citation on the first page. Copyrights for third-party componentsof this work must be honored. For all other uses, contact the Owner/Author.
Copyright is held by the owner/author(s).
UbiComp’14 Adjunct, September 13-17, 2014, Seattle, WA, USAACM 978-1-4503-3047-3/14/09.
http://dx.doi.org/10.1145/2638728.2638783

ACM Classification Keywords
H.5.m [Information interfaces and presentation (e.g.,HCI)]: Miscellaneous.

Introduction
Learning a new language can be quite frustrating,especially as the individual learner has often difficulties torecognize progress and to identify problem areas. Good

teachers are usually essential for success, however theyfrequently don’t have sufficient time to individualize theteaching experience towards the needs of individualstudents [4].
We wonder if there are ways to support the learning
process and give individualized feedback about thelearners performance. For the beginning we focus onreading and text comprehension. For reading
comprehension tasks, eye gaze seems an interesting
modality as it contains a lot of information about ourreading behavior and our cognitive status [5].
We explore, if and how implicit text annotations using eye
gaze can support second language learners and theirteachers. We start with giving readers quantified feedback
about their behavior, answering simple quantitative
questions: How fast did they read a paragraph? Howmuch re-reading did they do? Yet, finally we want to givethe reader feedback about their concentration and finallytext comprehension level.
In this paper, we show, how we can use eye gaze behavior
to annotate paragraphs with reading speed, the amount of
re-reading and aggregated fixation count in firstexperiments with 2 users and 2 texts. We present someideas about representing the different metrics andevaluate our initial annotation visualizations by presenting
them to 2 English teachers.

Approach
Figure 1: Experimental Setup:
User reading the text from an
English comprehension test. The
gaze patterns recorded by thestationary eye-tracker for the firsttext (mid for the student withlower English skills, bottomhigher skills).

We choose reading comprehension for English learners, as
English is the most common language people study andwe have easy access to students with varying levels atuniversity. In general, we want to explore three basic typesof annotations:

Word/Sentence level– we want to mark difficult words orsentence structures, keeping a list of unknown words andgive students hints on how to better understandgrammatical constructs. This might be the most difficult
task, due to accuracy problems with current eye-tracking
technology.
Paragraph-based– to give students feedback about theirreading progress and how well they read a given text, we
assume paragraph based analysis is useful and possibleusing even commercially available devices for the end user.
Reading Flow – Tracking a the high level ”reading
sequence” can give further insights into thecomprehension skills of the learner, especially during texts
assessing the reading comprehension of the learner withquestions, e.g. the sequence between gazing on questions
and paragraphs can give insights on how well the reader
understands the question.
In this paper, we focus on paragraph based annotations
only, as we think they already can give valuable support tothe learner and are feasible to implement with our current
technology. We select the following annotations for astart. The annotations are inspired by lab internal
discussions, some of the related work and are subject tochange [3].
We highlight reading speed by background color andintensity, slow speed with darker hue, faster speed withlighter hue. Reading speed is given by how long theparticipants eye gaze is in a paragraph region.
We estimate the amount of re-reading by comparing theline count of the paragraph with estimating the line countby using eye gaze using a method from Kunze et al. [7].

The amount of re-reading is shown by a arrow pointing
back up.
We aggregate fixations in larger fixation areas applying afiltering method from Busher et al. [3]. The number offixation areas are shown as a eye icon next to theparagraph.
Figure 2: Text 1: Eye-gazeannotated document for theparticipant with lower English
skills. Shading shows the reading
speed: the darker the slower.
The arrows on the right show theamount of re-reading and the sizeof the eye next to the paragraphthe number of fixation areas.

Figure 3: Text 1: Eye-gazeannotated document for theparticipant with higher English
skills.

milliseconds, an estimate for the amount of re-reading(estimated line numbers based on eye gaze minus real linenumbers), reading speed in seconds, and how manydifficult words and sentences the participant marked. Weuse these statistics to generate the annotated documents.
You see the result for Text1 in Figures 2 and 3.
Table 1: Summary for participant 1 (higher score)

Experimental Setup
2 subjects (Japanese male, undergraduate students) took
part in the experiment. One person is a high score holderof TOEIC, and the other has a low score. We used 2
English documents from standard comprehension tests as
we focus on second language learning skills. An easy text
from TOIEC and a more difficult text from SAT. To getthe gaze data, we used stationary eye tracker SMI RED
250. The eye gaze is sampled with 250 Hz.
First of all, we performed an initial calibration andchecked the calibration went well or not. And then, weasked subjects to read an English document. Documents
are projected on display of stationary eye tracker. Subjects
can read over as many times as they like (see Figure ??).
After reading, we asked them to solve questions related
the document. At this time, both the document and thequestions have projected, and the subjects can solve
questions by interactive looking. Afterwards we asked
them to put marks on sentences and words which theyfound difficult and to answer questionnaire about
document’s difficulty. We performed 2 sets of proceduresfrom calibration to questionnaire.

Initial Results
Tables 1 and 2 show the estimated metrics for each of thesubjects: The paragraph id with number of lines, thenumber of fixation areas, the average fixation duration inParagraphnum of lines
fixation areas
fixation duration
re-reading
reading speed
difficult words
difficult sentences

Text1(easier)123434346
1457
17.6 38.3 21.1 25.5
0.5
1.5
10.5
7.8
9.6
7.7
7.9
1
1000000

Text2(difficult)
123
78826
33
32
64.6 62.5 63.0
2.5
2
1.5
9.2
7.8
7.9
96
110
1
1

Table 2: Summary for participant 2 (lower score)

Paragraphnum of lines
fixation areas
fixation duration
re-reading
reading speed
difficult words
difficult sentences

Text1(easier)
12
3
4
3
4
3
4
26
46
37
19
55.3 88.1 44.3 33.6
22
1.5
2.5
24.6 22.0 16.1 10.3
02
1
10
1
1
1

Text2(difficult)123788231686.5 89.3
2410.8 11.2
101114120

We showed the resulting visualizations to two professional
English teachers, both could easily identify the averagelevel of the students given the annotated document

(without detailed explanation).In addition, they found theannotations useful to assess students and would like to
use the system in a class setting to get more individual
feedback about the skill level of their students.
They mentioned that the re-reading count per paragraphcould be potentially very useful to assess the language
skill of Japanese English learners. As some Japanese
students with lower English skills seem to reconstruct theJapanese word order of the sentence. This holds also with
our observations, trying to assess language expertise skill
directly from eye gaze. One teacher did not like the eye
icon next to the paragraphs. Additionally, both havestrong interest in reading flow visualizations.

Related Work
Reading is a well studied subject in the cognitive sciences.
However, there are very few researchers recognizing andevaluating reading behavior in realistic settings [2].
Busher et al. discuss in general the feasibility of gazebased annotations for documents [3]. There are also someefforts to infer the users expertise and language skill using
eye tracking [6]. Beidert et al. present the text 2.0
framework to adjust the reading experience dynamicallydepending on user’s eye gaze [1]. We are unaware of any
research that uses gaze based annotations to support
second language learning.

Conclusion and Future Work
Although the initial results are encouraging, especially inregard to the expert feedback from the English teachers,
we need to record more data and see if we can find therelation between English comprehension level andresulting annotations. Eye gaze can be influenced by a lotof parameters.
Additionally, if we are successful with the stationary setup,

we plan to evaluate our technology in more real-life
setups, first using a mobile eye tracker together with ourdocument image retrieval technology to associate eyegaze to the document, second using tablets and eye gazeestimation on them using only the camera. sReferences[1] Biedert, R., Buscher, G., Schwarz, S., Hees, J., andDengel, A. Text 2.0. In WIP CHI ’10 (2010).
[2] Bulling, A., Ward, J. A., Gellersen, H., and Troster, G.
Eye movement analysis for activity recognition using
electrooculography. IEEE Transactions on PatternAnalysis and Machine Intelligence 33, 4 (2011),
741–753.
[3] Buscher, G., Dengel, A., van Elst, L., and Mittag, F.
Generating and using gaze-based document
annotations. In CHI ’08 Extended Abstracts on
Human Factors in Computing Systems, CHI EA ’08,
ACM (New York, NY, USA, 2008), 3045–3050.
[4] Genesee, F., Lindholm-Leary, K., Saunders, W., andChristian, D. English language learners in us schools:
An overview of research findings. Journal of Educationfor Students Placed at Risk 10, 4 (2005), 363–385.
[5] Kunze, K., Iwamura, M., Kise, K., Uchida, S., andOmachi, S. Activity recognition for the mind: Towarda cognitive quantified self. Computer 46, 10 (2013),0105–108.
[6] Kunze, K., Kawaichi, H., Yoshimura, K., and Kise, K.
Towards inferring language expertise using eye
tracking. In CHI’13 Extended Abstracts on Human
Factors in Computing Systems, ACM (2013), 217–222.
[7] Kunze, K., Kawaichi, H., Yoshimura, K., and Kise, K.
The wordometer–estimating the number of words readusing document image retrieval and mobile eye
tracking. In ICDAR, IEEE (2013), 25–29.