Eye Gaze-based Unknown Word Detectionin Non-native Language Reading using SVMs and Random Forests
Rui Hiraoka, Hiroki Tanaka, Sakriani Sakti, Koichiro Yoshino, Graham Neubig, and Satoshi Nakamura
Abstract— This paper proposes machine-learning based
method to detect unknown words during natural reading byusing eye-tracking features. A previous approach [1] utilizes
gaze duration and word rarity features for detection. However,the performance of the previous approach is not sufﬁcientduring natural reading. To improve detection performance, wetry to 1) apply support vector machines (SVM) [2] and Random
Forests (RF) [3], and 2) use novel eye movement features thatwere not considered in the previous work. The experimental
results demonstrate that SVM and proposed eye movement
features are capable of improving detection performance asmeasured by F-measure.

I. INTRODUCTION
Eye movement has been considered as major indicatorof human reading strategies [4], [5]. An unknown word
detection algorithm based on total gaze duration and word
rarity was proposed by Hyrskykari [1], who used a wordunit threshold function calculated using word frequency
counts from a large corpus1 . However, the system requires
training of users to achieve sufﬁcient detection performance.
Therefore it is not effective for natural reading by untrainedusers. This paper proposes a new unknown word detection
method which focuses on optimization for natural reading
by untrained users.
II. P ROPOSED METHOD
We examined various features including word rarity, word
length, and three eye gaze features: ﬁrst gaze duration, maxgaze duration, and total gaze duration. The contribution ofeach feature was evaluated by the correlation coefﬁcientwith unknown-known label as well as the p-value on a twoway t-test between those labels. Moreover, we utilized two
well-known machine learning methods, RF and SVM withradial basis function (RBF) kernels, as classiﬁers on the mosteffective eye gaze features.

TABLE IF EATURE SELECTION RANKING BY CORRELATION COEFFICIENT r.
FeatureWord rarity: WR
Max gaze duration: gmax
Word length: WLFirst gaze duration: gﬁrst
Total gaze duration: gtotalr0.499
0.3750.353
0.336
0.305

p-value on t-test.x10−151.x10−28.x10−94.x10−20.x10−24

TABLE II
10- FOLD C ROSS VALIDATION C LASSIFICATION S CORES .
MethodPrecision
RecallFalse Positive
F-measureFeatures

Baseline
0.612
0.412
0.016
0.493
WR
gtotal

RF
0.544
0.466
0.026
0.502
WR
gtotal

RF
0.594
0.523
0.024
0.556
WR
gmax

SVM
0.485
0.763
0.050
0.593
WR
gtotal

SVM
0.520
0.703
0.041
0.598
WR
gmax

out of 4462 words were annotated as unknown in total. Table
I shows the ranking of each feature contribution. Max gaze
duration was the most effective eye gaze feature as measured
by correlation coefﬁcients and p-values. Table II shows
comparison of classiﬁers and eye gaze features by 10-foldcross validation. As a baseline, we considered the threshold
function [1]. The results shows that the proposed methods
using RF and SVM with max gaze duration features improve
performance as measured by F-measure. The performanceachieved by SVM was signiﬁcantly improved compared withthe baseline threshold function and RF (p < 0.01).
IV. C ONCLUSION
We proposed several eye gaze features and classiﬁers todetect unknown words. Using the most effective eye gazefeatures and SVM with RBF kernel, we achieved a signiﬁcant
improvement in detection F-measure from the 0.493 baseline
to 0.598.

III. E XPERIMENTAL E VALUATIONR EFERENCES

We recorded eye movement data from 5 Japanese speakers
naturally reading English. As the eye-tracker, we used aTobii pro x2-30. Each participant read 600-1000 words andannotated whether each word was unknown or known. 279[1] A. Hyrskykari, Eyes in attentive interfaces: Experiences from creatingiDict, a gaze-aware reading aid. Tampereen yliopisto, 2006.
[2] C. Cortes and V. Vapnik, “Support-vector networks,” Machine learning,
vol. 20, no. 3, pp. 273–297, 1995.
[3] L. Breiman, “Random forests,” Machine learning, vol. 45, no. 1, pp.
5–32, 2001.
[4] P. Martı́nez-Gómez and A. Aizawa, “Recognition of understanding
level and language skill using measurements of reading behavior,” inProceedings of the 19th international conference on Intelligent UserInterfaces. ACM, 2014, pp. 95–104.
[5] K. Kunze, H. Kawaichi, K. Yoshimura, and K. Kise, “Towards inferring
language expertise using eye tracking,” in CHI’13 Extended Abstracts
on Human Factors in Computing Systems. ACM, 2013, pp. 217–222.

*This work was supported by JSPS KAKEN 26540117 and 16K16172.
R. Hiraoka, H. Tanaka, S. Sakti, K. Yoshino, G. Neubig and S.
Nakamura are with Graduate School of Information Science, Nara Institute of Science and Technology, Takayama-cho 8916-5, Ikoma-shi, Nara,
Japan {hiraoka.rui.hj9, hiroki-tan, ssakti, koichiro,neubig, s-nakamura}@is.naist.jp
1 British National Corpus (2005). Oxford University Computing Services,at http://www.natcorp.ox.ac.uk/