Personalized Unknown Word Detectionin Non-native Language Reading using Eye Gaze
Rui Hiraoka

Hiroki Tanaka

Sakriani Sakti

Nara Institute of Science andTechnology, Japan

Nara Institute of Science andTechnology, Japan

Nara Institute of Science andTechnology, Japanhiraoka.rui.hj9@is.naist.jp
hiroki-tan@is.naist.jp
ssakti@is.naist.jp
Graham Neubig
Satoshi Nakamura
Nara Institute of Science and∗Technology, Japanneubig@is.naist.jp
ABSTRACTThis paper proposes a method to detect unknown words
during natural reading of non-native language text by using
eye-tracking features. A previous approach [5] utilizes gazeduration and word rarity features to perform this detection.
However, while this system can be used by trained users,
its performance is not sucient during natural reading byuntrained users. In this paper, we 1) apply support vector
machines (SVM) [1] with novel eye movement features thatwere not considered in the previous work and 2) examine theeect of personalization. The experimental results demonstrate that learning using SVMs and proposed eye movement
features improves detection performance as measured by Fmeasure and that personalization further improves results.

CCS Concepts

Nara Institute of Science andTechnology, Japans-nakamura@is.naist.jp
allowing for more ecient and appropriate display of translations and explanations to help users reading non-nativelanguage.

We focused on eye-gaze input to achieve rapid

and eortless system. Eye tracking allows a user to read adocument in their non-native language without using hands
e.g. keyboard or mouse. Therefore, one of the benets of use
eye-gaze based interfaces is that they are faster than using
a mouse, as many studies show [14][13]. Hyrskykari [5] hasproposed a gaze-based unknown word detection approachthat utilizes total gaze duration, word rarity, and a simple threshold function. After user training, their approachachieve high detection performance for unknown words (recall : 91.0%, false positive : 2.4%). However, without user
training, it had signicantly lower detection performance(recall : 28.8%, false positive : 0.9%). Therefore, it is noteective for untrained users.

This paper proposes a new

method for eye-gaze based unknown word detection during•Human-centered computing → Human computer interac-

natural reading that aims to achieve acceptable performancetion (HCI) ;even when used by untrained users. We extend Hyrskykari's
approach [5] by using machine learning, dening a number ofKeywords
Eye movement; unknown word detection; natural reading

1.

INTRODUCTION

new features, and selecting the most eective ones. We also
examined the eect of personalization (subject dependency).
The experimental evaluation conrmed the eectiveness of1) using a novel eye movement feature, max gaze duration,and 2) personalization to individual users.

Unknown words appear more often in non-native language
than native language, and thus unskilled non-native readersneed to use dictionaries and machine-translation software toassist themselves in reading. However, the performance ofmachine translation between many languages is not oftensucient for usage in this situation, and manual search ofdictionaries can be time-consuming. Our goal is to develop
an automatic and quick unknown word detection system,∗Now at Carnegie Mellon University, USA.

2. RELATED WORKThis work focuses on how to detect unknown words using
eye movement data while a reading non-native language.
For approaches without eye movement, an unknown word
detection method based on click logs was proposed by Ehara[3].

This approach uses a large click-log corpus to predict

unknown words. In contrast, eye movement has been considered as a major indicator of human reading strategies.
Rayner claried the relation between eye movement and unknown words showing that the duration of xation tends tobe longer on low-frequency words and the number of regressive eye movements tends to increase.

Kunze and Gomez

proposed an algorithm to estimate language skills from eye
movement [8][9]. This work showed xation tends to occur
frequently on dicult words.

To the authors' knowledge,

Hyrskykari has proposed the only gaze based method toautomatically detect unknown words [5].

In this previouswork, she developed iDict, which is an application to detectunknown words and display their explanation. This paper
attempts to improve unknown words detection performanceby using other eye movement features and personalization.

3.

UNKNOWN WORD DETECTION
In this section, we explain Hyrskykari's work including

a description of its features and threshold-based detection
method.

3.1

Figure 1: Fixation mapping. sx+1 is the nearest coordinate
to fk so fk belongs to wx+1 . Therefore, w(k) = x + 1 in thisFeatures

example.

Eye movement indicators of human reading strategies andactivities are xation, saccade, and regression [10].

Fixa-tions are a type of eye movement that consists of staring

4. PROPOSED METHOD

Gazes are dened as the sum of all

In this paper, we propose a method for unknown wordxations on a certain region. Saccades are a quick, simul-detection that uses a classier with new eye gaze features,

taneous movement of the right and left eyes between twoand additionally personalize this to each individual user.

at a single location.

or more phases of xation.
movements.

Regressions are backward eye

Text documents are formed by a sequence of

4.1 Fixation mapping

N words, W = {w1 , w2 , ..., wN }. A sequence of xations
F = {f1 , f2 , ..., fM } is given from recording of eye movement data. fk is a coordinate on the target screen. Eachulius document. For each word, eye movement features arefeature is dened formally as follows:

include gaze duration, extracted pupil diameter size, andis consecutive sum of all xation durations which occurred on word wx prior to saccade to another word.

is total sum of all xation durations mapped to wordclassied automatically into xations and saccades by using
software provided with the eye tracker.
obtain the coordinates of wx 's start point sx and end pointxation fk focuses is based on Euclid distance between xationfk

rarity of word wx : rare(wx )For each word in the stimulus document, word rarity

1

is calculated in a large corpus . This is done by calculating the rank of word wx 's frequency in the corpus

rank(wk ) and rounding values less than 100 or morethan 6000 up and down respectively:(rank(wk ) < 100)
(rank(wk ) > 6000)(else).

xth word wx 's start point sx and end point ex
Fixation fk belongs to the word which has

respectively.

the minimum distance point out of all of the start and end
points. Figure 1 shows an example of xation mapping. The
reason why we use the edges of each word instead of the cenwere often large, so xations were often mapped inaccurately
when using the centers of short words.

(1)

4.2 Feature extraction
This section describes features that we use in the proposed

Threshold function

method.

thh − thl(rare(wx ) − 100),
100 − 6000

(2)th(wx ) is the total gaze duration threshold determining whether word wx is unknown or known. thh is thewhere,threshold of total gaze duration for high-frequency words(rare(wk ) ≤ 100) in the corpus.

thl is the threshold total
gaze duration for low-frequency words (rare(wk ) ≥ 6000) inthe corpus. thh , thl are dened as below.
thh = µh + ah σh , and thl = µl + al σl ,where µh and µl are the mean total gaze duration of highand low-frequency words in the corpus.

and theter is because the dierence between short and long words

Previous work uses a threshold function for the detection

Likewise, σh andσl are standard deviations, and ah , al are parameters tooptimize detection performance to a given data set (ah > 0,al > 0). The prediction is given whether gtotal (wx ) is larger
than th(wx ). We used this method as a baseline to comparewith our proposed method.

1

In general, eye tracking data is a sequence ofgaze points sampled at a certain sampling rate, which are

ex . An algorithm to determine the word xation on which

wx including regressive xations.

100
rare(wx ) = 6000
rank(w )kThese features

To map xations to the words in stimuli documents, wetotal gaze duration on word wx : gtotal (wx )th(wx ) = thh −extracted based on the mapped xations.
regressions.

gaze duration: g(wx )3.2

Fixations should be mapped to each word of the stim-

Specically previous work used British National Corpus(2005) at http://www.natcorp.ox.ac.uk/

rst gaze duration on word wx :is the rst gaze duration mapped on word wx . If wx
is skipped once, we considered the regressive gaze duration as the rst gaze duration.
number of xations n(wx ):total number of xations occurred on word wx .
number of regressions on word wx :total number of regressive xations occurred on wordwx .
reg(wx ) = { k ∈ [2, m] | w(k − 1) > x, w(k) = x } .
(3)w(k) is a function to determine on which word the kthxation is mapped. When w(k) = x, k th xation ismapped on the xth word.
mean xation duration on word wx :is averaged duration of all xations which are assigned
to word wx .

maximum gaze duration on word wx :gmax (wx ) = max {G(wx )} .

Table 1: Participants information. P: Participants' ID, UW:(4)

A sequence of gazes on word wx is given as G(wx ) =Number of unknown words, UW rate: unknown word rate
in experimental document, ET: time elapsed to read.

{g1 (wx ), g2 (wx ), ..., gL (wx )}.

pupil diameter variation on word wx :pvariation (wx ) = pmax (wx ) − pmin (wx ) .

(5)

pmax (wx ) is the maximum pupil diameter on word wx
and pmin (wx ) is the minimum pupil diameter.
word length: len(wx )Pixel length of word wx on a 1600x900 screen:len(wx ) = D(sx , ex ) .

(6)

UWUW RateET [s]

TOEIC

P1

720.076

1909.7475

P21220.1292418.9530

P3720.076

656.6

560

P4

910.0961309.0

675

P5

60

0.063

564.7

680

P6

750.079

551.8

685

P7

420.044

865.1

740

P8

360.037

918.0

745

P9

390.041

765.0

785

P10

310.033

460.8

790

Total

640

0.067

10419.5

where D(sx , ek ) is a function to calculate Euclid distance between the xth word wx 's start point sx andend point ex .

4.3

Feature selection

We rst attempted to use all features for detection but
some of the features were not as eective as we predicted. To
nd eective features to detect unknown words accurately,
we calculated correlation coecients and p-values according
to student's t-test for each of the eye movement and linguistic features. The t-test measured whether the mean valuesof unknown and known words were signicantly dierent.
According to the correlation coecients and t-test, we se-Figure 2: An example of the stimulus document.

lected eective features. We also normalized every feature
to reduce the dierence of units in each feature.

4.4

made an attempt to read and understand the material. AsClassifier

the visual stimuli, we used documents from Japanese en-

We used SVMs with RBF kernels as classiers. An eval-trance English examination book entitled sokudokueitango

2uation was performed by 10-fold cross validation (for per-by H.Kazahaya, TOEFL

sonalized models) and leave-one-subject-out cross validation

demic paper by Rotter [11]. The visual stimulus documents(for non-personalized models). As our evaluation measure,

were displayed on a screen with 6 line breaks to avoid verti-

we used F-measure because our recorded data was biased

cal confounds (Figure 2). Every document was displayed as(unknown samples:9350

a PDF le. There were less than 4 sentences in each page.

words). We used grid-search to nd optimized parameters

As the eye-tracker, we used a table-mounted, Tobii pro x2-

for each model.

30.

640 words and known samples:

sample examination, and an aca-To calculate word rarity, We used TreeTagger [12] toextract stems of verbs and nouns from each word. For those

5.

EXPERIMENTAL EVALUATION

5.1

Eye movement recording

stems, we extracted word frequency ranking from the BNC
word frequency lists [6], then we determined rare(wx ) basedon equation (3).

We recorded eye movement data while subjects read astimulus document on a screen. While reading on the screen,
the head-movements of participants were not conspicuous.
Therefore, we used a table-mounted eye tracker because itcan obtain focal points with high resolution accuracy whenhead movement is relatively small [2].

5.2

Procedure(9 males and 3 females) who can read English (assessed
Eachparticipant gave informed consent before the data recording.

We removed 2 participants who had fatal errors of the eye
tracker during recording and nally 640 out of 9490 words
were annotated as unknown.

5.3 Results

We recruited a total of 12 Japanese graduate students
by the TOEIC [15] English skill assessment score.).

Each participant read a total of 949 words and annotatedwhether each word was unknown or known on printed paper.

Demographics of all participants are shown in Table

1. The participants rst entered an experiment room, andread three types of English documents while the eye-tracker
recorded eye movement. After recording, participants weredirected to manually annotate unknown words. Participants
were asked to summarize the documents to ensure that they

5.3.1 Feature selection
Table 2 shows the ranking of the features according withcorrelation with the known/unknown tags. The results indicate that all null hypotheses are rejected (p < 0.01). Therefore, all features were related to word labels. Max gaze duration was the most correlated eye gaze feature as measured
by correlation coecients and p-values.

5.3.2 Classification2

http://www.ets.org/toe/Table 2: Feature selection ranking by correlation coecient

Table 4:r.

features: rare/gmax /len.

The eect of subject dependency for SVM withDependency

No

Yes

0.502

0.507

Recall

0.608

0.616

False Positive

0.038

0.043

F-measure

0.550

0.556

rp-value

Precision

word rarity : rare(w)0.462e-233

word length : len(w)

0.339

e-171

max gaze duration : gmax (w)

0.318

e-53

rst gaze duration :gfirst (w)

0.265

e-15

features and SVMs with RBF kernels, we achieved a signif-

total gaze duration :gtotal (w)

0.252

e-45

icant improvement in detection as measured by F-measure

number of xation :n(w)

0.208

e-40

mean xation : fmean (w)

0.153

e-31

pupil variation: pvariation (w)

0.136

e-19

regression : reg(w)

0.109

e-16

Featurefrom the 0.422 of a baseline to 0.556. SVMs with max gaze
duration, word rarity, and word length provided the highestperformance.

In addtion, max gaze duration can signi-cantly contribute to improve detection performance.
Moreover, we found personalization yet improves the detection performance. One limitation of our study is that, it
has shown that pupil size diameter change needs 200-300ms
latency to reect human perception [4].

Table 3 shows a comparison of classiers and eye gazefeatures according to 10-fold cross validation.

However, this la-tency was not considered in our approach.

Also, xation-As a base-saccade classication was performed by default settings ofline, we considered the threshold function as mentioned in

the Tobii studio IV-lter. Hence, we would like to consider

equation (4). The results showed that the proposed method

other appropriate xation-saccade classication algorithm

using SVM with max gaze duration and word length featuresin future work.

improved performance as measured by F-measure. We applied the paired-bootstrap [7] test to conrm statistical signicance, and we found that the improvement between thebaseline and proposed learning method using SVMs was asignicant (p< 0.01).

7. ACKNOWLEDGMENTS
This work was supported by KAKEN 26540117 and 16K16172.

In addition, use of max gaze du-ration provided signicant improvement (paired boot strap
test between SVM with total gaze and SVM with max gaze:p < 0.01).

We also conrmed the SVMs with max gaze

duration, word rarity, and word length had the best performance (F-measure: 0.556).

We conrmed that results re-main at 0.556 by randomizing the folds several times (0.556± 0.001).

8. REFERENCES[1] C. Cortes and V. Vapnik. Support-vector networks.
Machine learning, 20(3):273297, 1995.
[2] A. Duchowski. Eye tracking methodology: Theory andpractice, volume 373. Springer Science & BusinessMedia, 2007.
[3] Y. Ehara, N. Shimizu, T. Ninomiya, and H. Nakagawa.
Personalized reading support for second-language web

Table 3: Classication results.

documents by collective intelligence. In Proceedings ofrare
gtotalrare
gtotal

rare
len

rare
gmax

rare
gmax
len

classier

Baseline

SVM

SVM

SVM

SVM

Precision

0.482

0.457

0.5010.472

0.507

inuence of gaze position on pupil size measurement:

Recall

0.375

0.630

0.581

0.639

0.616

analysis and correction. Behavior research methods,

False Positive

0.027

0.054

0.042

0.052

0.043

F-measure

0.422

0.530

0.538

0.543

0.556

Featuresthe 15th international conference on Intelligent userinterfaces, pages 5160. ACM, 2010.
[4] B. Gagl, S. Hawelka, and F. Hutzler. Systematic

43(4):11711181, 2011.
[5] A. Hyrskykari. Eyes in attentive interfaces:
Experiences from creating iDict, a gaze-aware reading
aid. Tampereen yliopisto, 2006.

5.3.3

Subject dependency

Next, we examined the eect of subject dependency using the best model with max gaze duration, word rarity,
and word length features had a best performance. The result of leave-one-subject-out cross validation was comparedto the result of 10-fold cross validation described in previous section.

Table 4 shows that with subject dependency,

the detection performance becomes slightly better (0.550 to0.556). This indicates personalization is eective to improvethe detection performance.

6.

HTTP:< http://www. kilgarri. co. uk/bnc-readme.
html>(accessed 24 Feb 2016), 1995.
[7] P. Koehn. Statistical signicance tests for machinetranslation evaluation. In Empirical Methods in
Natural Language Processing (EMNLP), pages
388395. Citeseer, 2004.
[8] K. Kunze, H. Kawaichi, K. Yoshimura, and K. Kise.
Towards inferring language expertise using eye
tracking. In Computer-Human Inter- action (CHI)'13
Extended Abstracts on Human Factors in Computing
Systems, pages 217222. ACM, 2013.

DISCUSSION[9] P. Martínez-Gómez and A. Aizawa. Recognition ofWe proposed several eye gaze features and classiers todetect unknown words.

[6] A. Kilgarri. BNC database and word frequency lists.

Using the most eective eye gazeunderstanding level and language skill using
measurements of reading behavior. In Proceedings ofthe 19th international conference on Intelligent User
Interfaces, pages 95104. ACM, 2014.
[10] K. Rayner. Eye movements in reading and information
processing: 20 years of research. Psychologicalbulletin, 124(3):372, 1998.
[11] J. B. Rotter. Social learning and clinical psychology.
1954.
[12] H. Schmid. Probabilistic part-of-speech tagging using
decision trees. In Proceedings of the international
conference on new methods in language processing,volume 12, pages 4449. Citeseer, 1994.
[13] L. E. Sibert and R. J. Jacob. Evaluation of eye gazeinteraction. In Proceedings of the SIGCHI conferenceon Human Factors in Computing Systems, pages
281288. ACM, 2000.
[14] C. Ware and H. H. Mikaelian. An evaluation of an eye
tracker as a device for computer input2. In ACM
SIGCHI Bulletin, volume 17, pages 183188. ACM,1987.
[15] P. E. Woodford. The test of english for internationalcommunication (TOEIC). 1980.