UBICOMP/ISWC ’16 ADJUNCT, SEPTEMBER 12-16, 2016, HEIDELBERG, GERMANY

Estimation of English skill with amobile eye tracker
Olivier Augereau
Osaka Prefecture University
1-1 Gakuen-cho, Naka, Sakai,
Osaka , JAPANaugereau.o@gmail.com

Hiroki Fujiyoshi
Osaka Prefecture University
1-1 Gakuen-cho, Naka, Sakai,
Osaka , JAPANfujiyoshi@m.cs.osakafu-u.ac.jp

Kai Kunze
Graduate School of Media
DesignKeio University, Japan
kai.kunze@gmail.com

Koichi Kise
Osaka Prefecture University
1-1 Gakuen-cho, Naka, Sakai,
Osaka , JAPAN
kise@cs.osakafu-u.ac.jp

Abstract
Learning a foreign language such as English is an important task for many people. The process of learning takes
time and it is important to have a simple way to evaluate theprogress of the skill. We propose a method to evaluate thereader’s English skill based on a mobile eye tracking system. The eye tracker captures the reader’s behavior while
reading a document. The front camera of the eye tracker
records the scene image that contains the read document.
By using a retrieval algorithm we can recognize the read
document and project the eye gaze data from the scene image to the document space. Then, some features related to
the reading and solving behavior on several documents arecomputed. As a first result, we show that the TOEIC scorecan be estimated with an error of 36.3 points.

Author Keywords
eye tracking; language estimation; LLAH.

ACM Classification Keywords
Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the Owner/Author. Copyright is held by the owner/author(s).
UbiComp/ISWC ’16 Adjunct, September 12-16, 2016, Heidelberg, Germany
ACM 978-1-4503-4462-3/16/09.
http://dx.doi.org/10.1145/2968219.2968275

1777

H.5.2 [User Interfaces]: Input devices and strategies

Introduction
Due to globalization, it is more and more important for nonnative English speakers to study English language. During the learning process, it is necessary for the learners
to know their current skill by taking English tests. One of

UBICOMP/ISWC ’16 ADJUNCT, SEPTEMBER 12-16, 2016, HEIDELBERG, GERMANY

the most famous English test in some Asian and European
countries is the TOEIC (Test of English for International
Communication) [2]. For example, in 2014, around 2.4 millions Japanese people took this test to estimate their language skill 1
Unfortunately, taking the TOEIC has several constraints.
The participants must go to a specific place at a specifictime to take the exam, pay the fee and wait around 1 monthafter taking the test to get the results.

Figure 1: The mobile eye tracker
can be worn as glasses. One
camera records the scene and twoothers record the eyes.

the authors is around 82% of error for the understanding
prediction.
In the same year, Copeland et al. proposed to estimate thereading comprehension based on eye gaze movements
by using an artificial neural network [3]. The misclassification rate is quite low, but the dataset is heavily unbalanced(most answers are good) and the result are equal or underthe baseline classifier.

This motivates us to propose a new method for estimating
the English skill by simply wearing a mobile eye tracker.
The eye tracker records the eye movements which reflect
the reader’s skill. We compute some features based on theeye movements and combine them with the number of goodanswers given by the subject to estimate the TOEIC score.
Figure 1 is an illustration of a mobile eye tracker.

In 2015, Yoshimura et al. proposed a method to classify the
TOEIC score in three classes: low, middle and high [9]. By
computing 33 features related to blinks, saccades, fixations,etc. the authors succeeded to classify the English skill of 11
readers in 3 classes with an accuracy of 90.9%. But if thenumber of classes if augmented from 3 to 4, the classification accuracy drop to 54.5%. So the skill can be estimatedonly roughly.

In our experiment, 9 Japanese students read 10 documents
and solve the associated questions. The first result showsthat, on average, the TOEIC score can be estimated by our
method with an error of 36.3 points, which is 3.7 % of thewhole range of the possible scores (from 10 to 990 points).

The main difference of our proposal is the use of a mobile
eye tracker. Furthermore, our analysis of the English skill isbased on the reading behavior when the subject read a text
and solve the related questions which allow us to estimate
the TOEIC score more accurately.

Related Work

Proposed method

In 2013, Kunze et al. [4] proposed a method for estimating
the English skill of a reader. The authors show that the average number of fixations and the standard deviation of thenumber of fixations is different depending on the reader’sskill.

Our proposed method consists of the 5 following consecutive steps:
1. Recording the eye gaze
2. Converting the eye gaze to the document coordinates

In 2014, Martinez-Gomez et al. proposed another method
to analyze the language skill (TOEIC or TOEFL score) ofa reader [6]. Unfortunately, the performance obtained by

3. Computing fixations and saccades
4. Extracting the features

1

http://www.toeic.or.jp/library/toeic data/toeic en/pdf/data/
TOEIC Program DAA.pdf

1778

5. Estimating the TOEIC score

SESSION: EYEWEAR

FixationsAverage duration
Total number
Sum of duration
Proportion of duration
SaccadesAverage length
Average speed
Average time
Sum of time
Sum of the speed
Sum of the distanceInverse of the average time
Table 1: 11 features computed forthe 3 areas (main text, questions,and the overall document). The“proportion of duration” represents
the ratio of the sum of duration
divided by the total time spend. To
these 33 features we add 2 otherfeatures: the number of transitions
between the document and thequestions, and the number of goodanswers. In total, 35 features are
used.

Figure 3: The eye gaze is recorded in the scene image. We apply
LLAH to retrieve the document image and compute thegeometrical transformation.
Figure 2: The subject is wearing a mobile eye tracker while
reading a document.

Recording the eye gaze
The mobile eye tracker contains 3 cameras. Two are located inside the glasses and record each eye. The last one
is a front camera recording the scene image. The eye gaze
positions are computed using the data from the two eye
cameras, and the position is displayed on the scene image
captured by the front camera. Figure 2 represents a subject
wearing a mobile eye tracker while reading a document.
Converting the eye gaze to the document coordinates
By using a mobile eye tracker, the subject can read freely
a paper document. The position of the eye gaze is given
for the scene image but the document in the scene image
can have a perspective distortion. Before analyzing the eyegaze, we need to compute the geometrical transformationof the document in order to project the eye gaze position
from the scene to the document coordinates. Figure 3 illustrates this principle.

1779

In order to solve this problem, we use the LLAH ( Locally
Likely Arrangement Hashing ) algorithm [8]. It is a document image retrieval algorithm. An example of each document image to retrieve is stored in a database. Then, if oneof these documents appears on the camera, the document
will be retrieved and the geometrical transformation computed. After this step, the eye gaze positions are converted
in the document coordinates.
Computing fixations and saccades
The eye gaze movement of the reading behavior is composed of fixations (the position where the reader stop briefly
to acquire information) and saccades (the jumps between
the fixations) [7]. The Buscher algorithm [1] is used to compute the fixations and saccades from the raw eye gazedata. The characteristics of fixations and saccades are different depending on the reader’s skill.
Extracting the features
As displayed in the Fig. 4 the document is divided in threeareas corresponding to the right page, the left page and the

UBICOMP/ISWC ’16 ADJUNCT, SEPTEMBER 12-16, 2016, HEIDELBERG, GERMANY

leave-one-out subject cross validation.
Then, the ridge regression method [5] is used to estimate
the TOEIC score of each subject as a function of 35 features.

Experiment

Figure 4: The document is divided into 3 areas: the main text
area, the question area and the overall document area. Somefeatures are computed for each area.

combination of both areas. The left page contains the main
text and the right page contains the questions. We extractfour features about fixations and seven features about saccades (see table 1 for details). These 11 features are computed for the three areas of the document. The document
contains fixations and saccades from the main text plus theones from the questions, plus the saccades going from one
to the other. Two other features are added to these 33 features: the number of correct answers given by the reader,
and the number of saccades between the main text and thequestions. A total of 35 features are used.
Estimating the TOEIC score
Before estimating the TOEIC score, a backward stepwisefeature selection is used to remove the features which are
not relevant. The feature selection is not applied for eachsubject independently but for all the subjects based on a1780

We asked nine subjects to read 10 documents. Each document consists of two pages: the left one contains the main
text and the right one contains four questions. The subject
is free to read and solve the question in the order he wants.
After processing five documents, the subject has a short
break. The mobile eye tracker used for the experiment is an“Eye tracking Glasses 2.0 (ETG2.0)” made by SensoMotoric Instruments (SMI). The eye tracker was calibrated one
time for each participant before starting the experiment.
The average age of the subjects is 22.4 (maximum: 24,
minimum: 20, and standard deviation: 1.16). All subjects
are Japanese males and university students. Seven are
undergraduate students and 2 are master students. The average of the TOEIC score of the subjects is 660 (maximum:
930, minimum: 435, and standard deviation: 154.1).

ResultFirst, the feature selection was applied. The backward stepwise feature selection selected 20 features among the 35
ones. For the evaluation of the selected features, we computed the correlation coefficient between the feature andthe TOEIC score. The features with the highest correlation
coefficient are: the number of correct answers (0.74), thesum of the saccade time of the question area (0.70), theaverage fixation duration of the overall area (0.66), and thesum of the saccade length of the overall area (0.66).
The table 2 shows the result of the TOEIC score estimation. For each participant, we compute the absolute er-

SESSION: EYEWEAR

Subject

Abs. Error

A
B
C
D
E
F
G
H
I
Average

87.0
86.9
1.52
28.1
42.9
24.1
9.38
24.5
21.9
36.3

Table 2: Estimation of the TOEIC
score.

ror between the real TOEIC score and the estimated one.
On average, the absolute mean error is 36.3 points, with astandard deviation of 46.5 points.

Conclusion and future work
We presented a new method for estimating the language
skill of a reader by using a mobile eye tracker. After retrieving the document in the scene image recorded in the front
camera, the eye gaze of the user is projected in the document space. Some features related to the reading behavior
are extracted and then, the TOEIC score of the subject canbe estimated.
Our main outlook is to try to reduce the number of needed
documents for estimating the English skill. For now, reducing the number of document greatly reduce the performance.

Acknowledgment
This work is supported in part by JST CREST and JSPS
KAKENHI Grant Numbers 25240028, 15K12172, and 1681207700.

REFERENCES
1. Georg Buscher and Andreas Dengel. 2009.
Gaze-based filtering of relevant document segments. In
International World Wide Web Conference (WWW).
20–24.
2. Kiyomi Chujo and Kathryn Oghigian. 2009. How many
words do you need to know to understand TOEIC,TOEFL & EIKEN? An examination of text coverage andhigh frequency vocabulary. Journal of Asia TEFL 6, 2
(2009), 121–148.
3. Leana Copeland, Tom Gedeon, and Sumudu Mendis.
2014. Predicting reading comprehension scores from1781

eye movements using artificial neural networks andfuzzy output error. Artificial Intelligence Research 3, 3(2014), p35.
4. Kai Kunze, Hitoshi Kawaichi, Kazuyo Yoshimura, andKoichi Kise. 2013. Towards inferring language expertiseusing eye tracking. In CHI’13 Extended Abstracts on
Human Factors in Computing Systems. ACM, 217–222.
5. Saskia Le Cessie and Johannes C Van Houwelingen.
1992. Ridge estimators in logistic regression. Applied
statistics (1992), 191–201.
6. Pascual Martı́nez-Gómez and Akiko Aizawa. 2014.
Recognition of understanding level and language skill
using measurements of reading behavior. In
Proceedings of the 19th international conference onIntelligent User Interfaces. ACM, 95–104.
7. Keith Rayner. 1998. Eye movements in reading andinformation processing: 20 years of research.
Psychological bulletin 124, 3 (1998), 372–422.
8. Kazutaka Takeda, Koichi Kise, and Masakazu Iwamura.
2011. Real-time document image retrieval for a 10
million pages database with a memory efficient andstability improved llah. In Document Analysis andRecognition (ICDAR), 2011 International Conference
on. IEEE, 1054–1058.
9. Kazuyo Yoshimura, Kai Kunze, and Koichi Kise. 2015.
The Eye as the Window of the Language Ability:
Estimation of English Skills by Analyzing Eye
Movement While Reading Documents. In Document
Analysis and Recognition (ICDAR), 2015 13th
International Conference on. 251–255.