Personalized Unknown Word Detectionin Non-native Language Reading using Eye Gaze
Rui Hiraoka

Nara Institute of Science andTechnology, Japan

Hiroki Tanaka

Nara Institute of Science andTechnology, Japan

Sakriani Sakti

Nara Institute of Science andTechnology, Japanhiraoka.rui.hj9@is.naist.jp
hiroki-tan@is.naist.jp
ssakti@is.naist.jp
Graham Neubig
Satoshi Nakamura
Nara Institute of Science and∗Technology, Japan

Nara Institute of Science andTechnology, Japans-nakamura@is.naist.jp

neubig@is.naist.jp

ABSTRACT

allowing for more eﬃcient and appropriate display of translations and explanations to help users reading non-native
language. We focused on eye-gaze input to achieve rapid
and eﬀortless system. Eye tracking allows a user to read adocument in their non-native language without using hands
e.g. keyboard or mouse. Therefore, one of the benefits of use
eye-gaze based interfaces is that they are faster than using
a mouse, as many studies show [14][13]. Hyrskykari [5] hasproposed a gaze-based unknown word detection approachthat utilizes total gaze duration, word rarity, and a simple threshold function. After user training, their approachachieve high detection performance for unknown words (recall : 91.0%, false positive : 2.4%). However, without user
training, it had significantly lower detection performance(recall : 28.8%, false positive : 0.9%). Therefore, it is not
eﬀective for untrained users. This paper proposes a new
method for eye-gaze based unknown word detection during
natural reading that aims to achieve acceptable performanceeven when used by untrained users. We extend Hyrskykari’s
approach [5] by using machine learning, defining a number of
new features, and selecting the most eﬀective ones. We also
examined the eﬀect of personalization (subject dependency).
The experimental evaluation confirmed the eﬀectiveness of1) using a novel eye movement feature, max gaze duration,and 2) personalization to individual users.

This paper proposes a method to detect unknown words
during natural reading of non-native language text by using eye-tracking features. A previous approach utilizes gaze
duration and word rarity features to perform this detection.
However, while this system can be used by trained users,
its performance is not suﬃcient during natural reading byuntrained users. In this paper, we 1) apply support vector machines (SVM) with novel eye movement features thatwere not considered in the previous work and 2) examine theeﬀect of personalization. The experimental results demonstrate that learning using SVMs and proposed eye movement
features improves detection performance as measured by Fmeasure and that personalization further improves results.

CCS Concepts•Human-centered computing → Human computerinteraction (HCI) ;Keywords
Eye movement; unknown word detection; natural reading

1. INTRODUCTION
Unknown words appear more often in non-native language
than native language, and thus unskilled non-native readersneed to use dictionaries and machine-translation software toassist themselves in reading. However, the performance ofmachine translation between many languages is not often
suﬃcient for usage in this situation, and manual search ofdictionaries can be time-consuming. Our goal is to develop
an automatic and quick unknown word detection system,

2.

RELATED WORKThis work focuses on how to detect unknown words using
eye movement data while a reading non-native language.
For approaches without eye movement, an unknown word
detection method based on click logs was proposed by Ehara[3]. This approach uses a large click-log corpus to predict
unknown words. In contrast, eye movement has been considered as a major indicator of human reading strategies.
Rayner clarified the relation between eye movement and unknown words showing that the duration of fixation tends tobe longer on low-frequency words and the number of regressive eye movements tends to increase. Kunze and Gomez
proposed an algorithm to estimate language skills from eye
movement [8][9]. This work showed fixation tends to occur
frequently on diﬃcult words. To the authors’ knowledge,
Hyrskykari has proposed the only gaze based method toautomatically detect unknown words [5]. In this previous
work, she developed iDict, which is an application to detect∗Now at Carnegie Mellon University, USA.
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from Permissions@acm.org.

ICMI’16, November 12–16, 2016, Tokyo, Japanc 2016 ACM. 978-1-4503-4556-9/16/11...$15.00
http://dx.doi.org/10.1145/2993148.2993167

66

unknown words and display their explanation. This paper
attempts to improve unknown words detection performanceby using other eye movement features and personalization.

3. UNKNOWN WORD DETECTION
In this section, we explain Hyrskykari’s work including
a description of its features and threshold-based detection
method.

Figure 1: Fixation mapping. sx+1 is the nearest
coordinate to fk so fk belongs to wx+1 . Therefore,w(k) = x + 1 in this example.

3.1 Features
Eye movement indicators of human reading strategies andactivities are fixation, saccade, and regression [10]. Fixations are a type of eye movement that consists of staring
at a single location. Gazes are defined as the sum of all
fixations on a certain region. Saccades are a quick, simultaneous movement of the right and left eyes between two
or more phases of fixation. Regressions are backward eye
movements. Text documents are formed by a sequence of
N words, W = {w1 , w2 , ..., wN }. A sequence of fixations
F = {f1 , f2 , ..., fM } is given from recording of eye movement data. fk is a coordinate on the target screen. Eachfeature is defined formally as follows:

4.

PROPOSED METHOD

In this paper, we propose a method for unknown word
detection that uses a classifier with new eye gaze features,
and additionally personalize this to each individual user.

4.1

Fixation Mapping

Fixations should be mapped to each word of the stimulius document. For each word, eye movement features are
extracted based on the mapped fixations. These features
include gaze duration, extracted pupil diameter size, andregressions. In general, eye tracking data is a sequence ofgaze points sampled at a certain sampling rate, which areclassified automatically into fixations and saccades by using
software provided with the eye tracker.
To map fixations to the words in stimuli documents, we
obtain the coordinates of wx ’s start point sx and end point
ex . An algorithm to determine the word fixation on which
kth fixation fk focuses is based on Euclid distance between
fixationfk and the xth word wx ’s start point sx and end
point ex respectively. Fixation fk belongs to the word which
has the minimum distance point out of all of the start andend points. Figure 1 shows an example of fixation mapping.
The reason why we use the edges of each word instead of thecenter is because the diﬀerence between short and long words
were often large, so fixations were often mapped inaccurately
when using the centers of short words.

gaze duration: g(wx )is consecutive sum of all fixation durations which occurred on word wx prior to saccade to another word.
total gaze duration on word wx : gtotal (wx )is total sum of all fixation durations mapped to word
wx including regressive fixations.
rarity of word wx : rare(wx )For each word in the stimulus document, word rarity
is calculated in a large corpus1 . This is done by calculating the rank of word wx ’s frequency in the corpus
rank(wk ) and rounding values less than 100 or morethan 6000 up and down respectively:⎧⎪(rank(wk ) < 100)⎨100rare(wx ) = 6000
(1)
(rank(wk ) > 6000)⎪⎩rank(w ) (else).
k

4.2

Feature Extraction

This section describes features that we use in the proposed
method.

3.2 Threshold Function

Previous work uses a threshold function for the detectionthh − thlth(wx ) = thh −(rare(wx ) − 100),(2)
100 − 6000

first gaze duration on word wx :is the first gaze duration mapped on word wx . If wx isskipped once, we considered the regressive gaze duration as the first gaze duration.

where, th(wx ) is the total gaze duration threshold determining whether word wx is unknown or known. thh is thethreshold of total gaze duration for high-frequency words(rare(wk ) ≤ 100) in the corpus. thl is the threshold total
gaze duration for low-frequency words (rare(wk ) ≥ 6000) in
the corpus. thh , thl are defined as below.

number of fixations n(wx ):total number of fixations occurred on word wx .
number of regressions on word wx :total number of regressive fixations occurred on wordwx .
%%reg(wx ) = %{ k ∈ [2, m] | w(k − 1) > x, w(k) = x }% .
(3)w(k) is a function to determine on which word the kth
fixation is mapped. When w(k) = x, kth fixation ismapped on the xth word.

thh = µh + ah σh , and thl = µl + al σl ,where µh and µl are the mean total gaze duration of highand low-frequency words in the corpus. Likewise, σh andσl are standard deviations, and ah , al are parameters to
optimize detection performance to a given data set (ah > 0,
al > 0). The prediction is given whether gtotal (wx ) is larger
than th(wx ). We used this method as a baseline to comparewith our proposed method.

mean fixation duration on word wx :is averaged duration of all fixations which are assigned
to word wx .

1

Specifically previous work used British National Corpus(2005) at http://www.natcorp.ox.ac.uk/

67

maximum gaze duration on word wx :gmax (wx ) = max {G(wx )} .

Table 1: Participants information. P: Participants’
ID, UW: Number of unknown words, UW rate: unknown word rate in experimental document, ET:time elapsed to read.
UW UW RateET [s]
TOEIC
P1
72
0.076
1909.7
475
P2
122
0.129
2418.9
530
P3
72
0.076
656.6
560
P4
91
0.096
1309.0
675
P5
60
0.063
564.7
680
P6
75
0.079
551.8
685
P7
42
0.044
865.1
740
P8
36
0.037
918.0
745
P9
39
0.041
765.0
785
P10
31
0.033
460.8
790
Total 640
0.067
10419.5(4)

A sequence of gazes on word wx is given as G(wx ) ={g1 (wx ), g2 (wx ), ..., gL (wx )}.
pupil diameter variation on word wx :pvariation (wx ) = pmax (wx ) − pmin (wx ) .

(5)

pmax (wx ) is the maximum pupil diameter on word wx
and pmin (wx ) is the minimum pupil diameter.
word length: len(wx )Pixel length of word wx on a 1600x900 screen:len(wx ) = D(sx , ex ) .

(6)

where D(sx , ek ) is a function to calculate Euclid distance between the xth word wx ’s start point sx andend point ex .

4.3 Feature Selection
We first attempted to use all features for detection but
some of the features were not as eﬀective as we predicted. To
find eﬀective features to detect unknown words accurately,
we calculated correlation coeﬃcients and p-values according
to student’s t-test for each of the eye movement and linguistic features. The t-test measured whether the mean valuesof unknown and known words were significantly diﬀerent.
According to the correlation coeﬃcients and t-test, we selected eﬀective features. We also normalized every feature
to reduce the diﬀerence of units in each feature.

Figure 2: An example of the stimulus document.

4.4 Classifier

We recorded eye movement data while subjects read astimulus document on a screen. While reading on the screen,
the head-movements of participants were not conspicuous.
Therefore, we used a table-mounted eye tracker because it
can obtain focal points with high resolution accuracy whenhead movement is relatively small [2].

made an attempt to read and understand the material. As
the visual stimuli, we used documents from Japanese entrance English examination book entitled “sokudokueitango”
by H.Kazahaya, TOEFL2 sample examination, and an academic paper by Rotter [11]. The visual stimulus documents
were displayed on a screen with 6 line breaks to avoid vertical confounds (Figure 2). Every document was displayed asa PDF file. There were less than 4 sentences in each page.
As the eye-tracker, we used a table-mounted, Tobii pro x230. To calculate word rarity, We used TreeTagger [12] toextract stems of verbs and nouns from each word. For thosestems, we extracted word frequency ranking from the BNC
word frequency lists [6], then we determined rare(wx ) basedon equation (3).
Each participant read a total of 949 words and annotatedwhether each word was unknown or known on printed paper.
We removed 2 participants who had fatal errors of the eye
tracker during recording and finally 640 out of 9490 words
were annotated as unknown.

5.2 Procedure

5.3

We used SVMs with RBF kernels as classifiers. An evaluation was performed by 10-fold cross validation (for personalized models) and leave-one-subject-out cross validation(for non-personalized models). As our evaluation measure,
we used F-measure because our recorded data was biased(unknown samples: 640 words and known samples: 9350
words). We used grid-search to find optimized parameters
for each model.

5. EXPERIMENTAL EVALUATION
5.1 Eye movement Recording

We recruited a total of 12 Japanese graduate students(9 males and 3 females) who can read English (assessedby the TOEIC [15] English skill assessment score.). Eachparticipant gave informed consent before the data recording. Demographics of all participants are shown in Table
1. The participants first entered an experiment room, andread three types of English documents while the eye-tracker
recorded eye movement. After recording, participants weredirected to manually annotate unknown words. Participants
were asked to summarize the documents to ensure that they

5.3.1

Results
Feature Selection

Table 2 shows the ranking of the features according withcorrelation with the known/unknown tags. The results indicate that all null hypotheses are rejected (p < 0.01). Therefore, all features were related to word labels. Max gaze duration was the most correlated eye gaze feature as measured
by correlation coeﬃcients and p-values.
268

http://www.ets.org/toefl/

Table 2: Feature selection ranking by correlationcoeﬃcient r.
Featureword rarity : rare(w)word length : len(w)max gaze duration : gmax (w)first gaze duration :gfirst (w)total gaze duration :gtotal (w)number of fixation :n(w)mean fixation : fmean (w)pupil variation: pvariation (w)regression : reg(w)

r
0.462
0.339
0.318
0.265
0.252
0.208
0.153
0.136
0.109

Table 4: The eﬀect of subject dependency for SVM
with features: rare/gmax /len.
Dependency
No
YesPrecision
0.502 0.507
Recall
0.608 0.616
False Positive 0.038 0.043
F-measure0.550 0.556

p-value
e-233
e-171
e-53
e-15
e-45
e-40
e-31
e-19
e-16

features and SVMs with RBF kernels, we achieved a significant improvement in detection as measured by F-measurefrom the 0.422 of a baseline to 0.556. SVMs with max gaze
duration, word rarity, and word length provided the highest
performance. In addtion, max gaze duration can significantly contribute to improve detection performance.
Moreover, we found personalization yet improves the detection performance. One limitation of our study is that, it
has shown that pupil size diameter change needs 200-300ms
latency to reflect human perception [4]. However, this latency was not considered in our approach. Also, fixationsaccade classification was performed by default settings of
the Tobii studio IV-filter. Hence, we would like to consider
other appropriate fixation-saccade classification algorithm infuture work.

5.3.2 Classification
Table 3 shows a comparison of classifiers and eye gazefeatures according to 10-fold cross validation. As a baseline, we considered the threshold function as mentioned inequation (4). The results showed that the proposed method
using SVM with max gaze duration and word length features
improved performance as measured by F-measure. We applied the paired-bootstrap [7] test to confirm statistical significance, and we found that the improvement between thebaseline and proposed learning method using SVMs was asignificant (p < 0.01). In addition, use of max gaze duration provided significant improvement (paired boot strap
test between SVM with total gaze and SVM with max gaze:p < 0.01). We also confirmed the SVMs with max gaze
duration, word rarity, and word length had the best performance (F-measure: 0.556). We confirmed that results remain at 0.556 by randomizing the folds several times (0.556± 0.001).
Features

Table 3: Classification results.
rare
rare
rare
rare
gtotalgtotallen
gmax

classifier
Precision
RecallFalse Positive
F-measure

Baseline
0.482
0.375
0.027
0.422

SVM
0.457
0.630
0.054
0.530

SVM
0.501
0.581
0.042
0.538

SVM
0.472
0.639
0.052
0.543

7.

ACKNOWLEDGMENTS
This work was supported by KAKEN 26540117 and 16K16172.

8.

REFERENCES[1] C. Cortes and V. Vapnik. Support-vector networks.
Machine learning, 20(3):273–297, 1995.
[2] A. Duchowski. Eye tracking methodology: Theory andpractice, volume 373. Springer Science & BusinessMedia, 2007.
[3] Y. Ehara, N. Shimizu, T. Ninomiya, and H. Nakagawa.
Personalized reading support for second-language web
documents by collective intelligence. In Proceedings ofthe 15th international conference on Intelligent userinterfaces, pages 51–60. ACM, 2010.
[4] B. Gagl, S. Hawelka, and F. Hutzler. Systematic
influence of gaze position on pupil size measurement:
analysis and correction. Behavior research methods,
43(4):1171–1181, 2011.
[5] A. Hyrskykari. Eyes in attentive interfaces:
Experiences from creating iDict, a gaze-aware reading
aid. Tampereen yliopisto, 2006.
[6] A. Kilgarriﬀ. BNC database and word frequency lists.
HTTP:< http://www. kilgarriﬀ. co. uk/bnc-readme.
html>(accessed 24 Feb 2016), 1995.
[7] P. Koehn. Statistical significance tests for machinetranslation evaluation. In Empirical Methods in
Natural Language Processing (EMNLP), pages
388–395. Citeseer, 2004.
[8] K. Kunze, H. Kawaichi, K. Yoshimura, and K. Kise.
Towards inferring language expertise using eye
tracking. In Computer-Human Inter- action (CHI)’13
Extended Abstracts on Human Factors in Computing
Systems, pages 217–222. ACM, 2013.
[9] P. Martı́nez-Gómez and A. Aizawa. Recognition ofunderstanding level and language skill using
measurements of reading behavior. In Proceedings of

rare
gmax
len
SVM
0.507
0.616
0.043
0.556

5.3.3 Subject Dependency
Next, we examined the eﬀect of subject dependency using the best model with max gaze duration, word rarity,
and word length features had a best performance. The result of leave-one-subject-out cross validation was comparedto the result of 10-fold cross validation described in previous section. Table 4 shows that with subject dependency,
the detection performance becomes slightly better (0.550 to
0.556). This indicates personalization is eﬀective to improvethe detection performance.

6. DISCUSSION
We proposed several eye gaze features and classifiers todetect unknown words. Using the most eﬀective eye gaze69

the 19th international conference on Intelligent User
Interfaces, pages 95–104. ACM, 2014.
[10] K. Rayner. Eye movements in reading and information
processing: 20 years of research. Psychological bulletin,
124(3):372, 1998.
[11] J. B. Rotter. Social learning and clinical psychology.
1954.
[12] H. Schmid. Probabilistic part-of-speech tagging using
decision trees. In Proceedings of the international
conference on new methods in language processing,volume 12, pages 44–49. Citeseer, 1994.

[13] L. E. Sibert and R. J. Jacob. Evaluation of eye gazeinteraction. In Proceedings of the SIGCHI conferenceon Human Factors in Computing Systems, pages
281–288. ACM, 2000.
[14] C. Ware and H. H. Mikaelian. An evaluation of an eye
tracker as a device for computer input2. In ACM
SIGCHI Bulletin, volume 17, pages 183–188. ACM,
1987.
[15] P. E. Woodford. The test of english for international
communication (TOEIC). 1980.

70