JAIST Repository
https://dspace.jaist.ac.jp/

Title

Analysis of Eye Movements based on the Entropy of
the N‑gram Model for the Investigation ofJapanese Reading Processes

Author(s)Tera, Akemi; Shirai, Kiyoaki; Yuizono, Takaya;
Sugiyama, Kozo

CitationIssue Date

2007‑11Type

Conference Paper

Text version

publisher

URL

http://hdl.handle.net/10119/4072

RightsDescription

The original publication is available at JAIST
Press http://www.jaist.ac.jp/library/jaist‑press/index.html, KICSS 2007 : The Second
International Conference on Knowledge,
Information and Creativity Support Systems :PROCEEDINGS OF THE CONFERENCE, November 5‑7,
2007, [Ishikawa High‑Tech Conference Center,
Nomi, Ishikawa, JAPAN]

Japan Advanced Institute of Science and Technology

Analysis of Eye Movements based on the Entropy of the N-gram Modelfor the Investigation of Japanese Reading Processes
Akemi Tera†

Kiyoaki Shirai‡ Takaya Yuizono† Kozo Sugiyama††School of Knowledge Science
Japan Advanced Institute of Science and Technology‡School of Information Science
Japan Advanced Institute of Science and Technology{ tera, kshirai, yuizono, sugi }@jaist.ac.jp

Abstract
In order to investigate reading processes of Japanese language learners, we have conducted anexperiment to record eye movements during Japanese text reading using an eye-tracking system.
We showed that Japanese native speakers use“forward and backward jumping of eye movements” frequently [12] [13]. In this paper, weanalyzed further the same eye tracking data. Our
goal is to examine whether Japanese learners fix
their eye movements at boundaries of linguistic
units such as words, phrases or clauses when
they start or end “backward jumping”. Linguistic
boundaries are empirically defined based on theentropy of the N-gram model. Another goal is to
examine the relation between the entropy of the
N-gram model and the depth of syntactic structures of sentences. Our analysis shows that (1)Japanese learners often fix their eyes aroundlinguistic boundaries, (2) the average of the entropy is the greatest at the fifth depth of syntactic
structures.
Keywords: eye tracking, saccade, fixation,
N-gram model, entropy

1 Introduction
In the field of Japanese education, many Japanese language learners (Japanese learners) feelthat a Japanese text is difficult to read. The reasons are generally summarized as follows: there
is no space between words, a Japanese text consists of four different kinds of characters (Hiragana, Katakana, Kanji and Roma-ji), and it isdifficult to look up words represented by Kanji
in a dictionary since Kanji words have several
possible readings. In order to support the learning of Japanese, an investigation of Japaneselearners’ reading processes plays an importantrole.

Several studies aiming at analyzing reading
processes of Japanese learners have been conducted. Tera et al. [10] reported that
“Na-adjectives”, “sahen” and complex verbs areoften consulted by Japanese learners via a reading support system developed in [5] [6] [7] 1 .
They indicated that Japanese learners especiallywant to know information about predicates in
the Japanese text. From the beginning of 1990,
research on reading processes of Japanese or
English texts using an eye-tracking system were
initiated. Osaka [2] reported differences in thelocations where subjects fix their eyes according
to the types of characters in the text. Shigematsu
et al. [4] discussed differences of fixation ofeyes between Japanese and Chinese students.
We also tried to investigate reading processes
using an eye-tracking system [12] [13]. We
found that (1) when reading text in native or familiar languages, fixation time tended to be
short, (2) Japanese native speakers often showed“backward jumping of eye movements” and“forward jumping of eye movements”. It is wellknown that “backward jumping of eye movements” occurs when a reader finds out somediscrepancies while reading with respect to themeanings he holds, and is often used by skilledlanguage learners [1].
In this paper, we further analyze the eye tracking data obtained in [12][13]. We consider theentropy of the N-gram model, and examine thecorrelation between the entropy and the eye
movements.
The paper is organized as follows: in Section 2,
the entropy, fixation and saccade are briefly
introduced, since they are important concepts in
1“Na-adjective” and “sahen” are word classes inJapanese. The Na-adjective is one kind of adjective,
while sahen is a word which functions both as a noun
and as a verb.

this paper, and our goal is explained in moredetail. Section 3 describes the detailed procedures of our analysis. Results of our analysis arereported in Section 4. A preliminary survey ofthe relation between syntactic structures and entropy is presented in Section 5. We conclude thepaper in Section 6.

2 GoalWe here briefly introduce the entropy of the
N-gram model. The N-gram model is awell-known probabilistic language model widelyused in the research field of natural language
processing [9]. The N-gram model of charactersis the probabilistic model predicting the appearance of a character (ci) given its preceding N-1
characters (ci-N+1..ci-1). It is defined as follows:P(c i | c i" N +1 Lc i"1 ) (1)The N-gram model can be automatically trained
from a large amount of corpora [9]. Next, theentropy E of the N-gram model is given as be!
low:
E = "# P(c i | c i" N +1 Lc i"1 ) log P(c i | c i" N +1 Lc i"1 ) (2)
ci

!

!

In general, the entropy of the N-gram model isrelevant to linguistic boundaries such as
boundaries of words, phrases or clauses. For
example, let us consider the cases where characters ci-N+1...ci are in a word. In such cases, only alimited number of possible characters would
appear after the string ci-N+1 ... ci-1. Thus the entropy would be low since the probabilistic distribution P(c i | c i" N +1 Lc i"1 ) is not uniform, or isskewed. On the other hand, if there is a linguistic
boundary between ci-1 and ci, various characters
could appear after the string ci-N+1...ci-1. In such!cases, the entropy would be high sinceP(c i | c i" N +1 Lc i"1 ) tends to be uniform. To
summarize, if the entropy of the N-gram model
is high, we can assume that there is a linguistic
boundary at a position between characters ci-1
and ci.
Next, we introduce “fixation” and “saccade” ineye movements. When we read a text, we use
our eyes to get information from the text. It isalready known that humans do not move theireyes smoothly or constantly while reading, but
repeat fixation and saccade. “Fixation” meansthat the eyeballs stop moving for the eye toglance at the same point for a while in order tocarefully read a text, while “saccade” means thatthe eyeballs move from a fixation point to thenext fixation point. We call a saccade in theforward direction a “forward saccade”, while asaccade in the backward direction is a “backward saccade”.
The goal of this paper is to investigate thereading process of Japanese learners. More specifically, we investigate where backward saccades occur in the text. As described in Section
1, a backward saccade (backward jumping ofeye movement) occurs when a Japanese learner
cannot understand the text smoothly and wants
to read previous sentences again to make surethat what he/she understood before is correct. If
this is correct, we suppose that Japanese learners
would start or end a backward saccade notwithin a word but at linguistic boundaries. So we
propose the following hypothesis:[Hypothesis]
When reading a Japanese text, a backward
saccade starts or ends around linguistic boundaries.
Furthermore, as we mentioned before, linguistic boundaries can be empirically defined bymeasuring the entropy of the N-gram model. In
this paper, we will empirically prove the above
hypothesis through an analysis of real eyetracking data as well as the entropy of the
N-gram model trained from a large amount of
text.
Another goal is to examine the relation between the backward saccade and syntactic structures of sentences. We would like to know if
Japanese learners start or end backward saccades
at deep positions in syntactic structures. As apreliminary survey for this investigation, we
examine the correlation between the entropy of
the N-gram model and the depth of syntactic
structures.

3 Methodology
In this section, procedures to verify our hypothesis are described. The experiment for recording eye-tracking data is briefly described in
Subsection 3.1 [12][13], while the verification of
our hypothesis by use of the obtained
eye-tracking data is discussed in Subsection 3.2.
3.1 Collecting Eye Tracking Data
The following equipment is used to obtain
eye-tracking data of Japanese learners:
1) An eye-mark recorder EMR-8 (NAC)
2) A 44-degree lens3) An analyzing system (NAC)

4) A windows NT, 17 inches monitor
5) A head standSince subjects are required to have eyesightover 1.0 to record eye movements with oureye-mark recorder, an eyesight check is done
before the experiment. An illustration of the recording of the eye tracking data with the above
equipment is shown in Figure 1.
Figure 3. Saccades and Fixations on the Text

Figure 1. Experimental set-up
The procedures for obtaining eye-tracking dataconsist of the following 7 steps:
1) Choose intermediate learners as subjects
according to the results of the Standard
Japanese Ability Test and the English
TOEFL / TOEIC.
2) The explain on overview of the experiments
and instructions to the subjects, then letthem do preliminary experiments.
3) Ask the subjects to read texts displayed
randomly on a PC screen one by one. Timefor reading is not limited.
4) Record eye tracking data during reading.
5) Carry out an examination to check the subjects’ comprehension of texts, and obtain ascore for each subject.
6) Ask the subjects which texts are most difficult and easiest to understand, and the reason why.
7) Analyze the obtained eye tracking data as
well as other data (Questionnaires, etc.).

Figure 2. Japanese Text Used for the experiment

The Japanese text used in the experiment is anewspaper article excerpted from the Chunichi
Newspaper [11], shown in Figure 2. It contains
178 characters. Figure 3 shows a schematic representation of the obtained eye tracking data
overlapped with the text. In Figure 3, circles indicate fixations, while lines indicate saccades.
The size of the circles represents the time of the
fixations.
We collected eye-tracking data for 20 subjects.
They are classified into the following 5 groups
according to their native language and familiarity with Kanji [8]:
 Japanese-Native-Speaker(J)Subjects who are Japanese native speakers.
 Kanji-area(K)Subjects who use Kanji in their native language.
 Middle-area(M)Subjects who know Kanji but do not usually
use them.
 Non-Kanji-area-Asia(NA)Subjects who do not use Kanji and are fromAsia.
 Non-Kanji-area-Europe(NE)
Subjects who do not use Kanji and are from
Europe.
The Numbers of subjects in each group as wellas their nationalities are summarized in Table 1.
Table 1 Nationalities of Subjects
J
Japanese (4)K
China (5)M Korea (4)NA Nepal (1), Vietnam (1), Thailand (1)NE Belgium (1), Germany (1), Hungary(1), Spain (1)3.2 Analysis of Eye Tracking DataFirst, we train the N-gram model of characters.
We set N equal to 5, that is, we estimate a
5-gram model which predicts the probabilities ofappearance of characters given 4 preceding
characters. The model is trained by maximum
likelihood estimation from newspaper articles
published over 13 years. Then, the entropy of
the 5-gram model is calculated at all positions in
the text that subjects read in our experiment. The‘Position in the text’ refers to a position between
characters, and the entropy at a position between
ci-1 and ci is the entropy of the probability distribution P(c i | c i"4 c i"3c i"2c i"1) , where ci-4, ci-3, ci-2,
and ci-1 are the 4 characters appearing before that
position. Unfortunately, the entropy cannot becalculated
for all positions due to the data!
sparseness, since the entropy is not determinedwhen the string ci-4ci-3ci-2ci-1 does not occur in the
training corpus. Hereafter we call such positions“uncertain entropy positions”.
Next, we extract fixations at the start and at theend of backward saccades from the eye tracking
data. The positions of extracted fixations in the
text area should be identified, since our current
goal is to examine if fixations happen at linguistic boundaries or at high entropy positions.
However, it is rather difficult to decide the exact
positions of fixations. Osaka [2] reported thatreaders saw between 2 and 5, mostly 3 and 4
characters when they fixed their eyes. Thismeans that Japanese learners may see not a point
but an area including several characters at fixations. Therefore, we suppose that subjects see 3
or 4 characters when they fix their eyes, andidentify characters they glance at fixations asshown in Figure 4.
Figure 4 Fixation Area
In this figure, a star shows for the center of eye
glance at fixation, while ‘C’ stands for characters supposed to be seen by subjects. That is,when the center of eye glance at fixation is on acharacter, we suppose that subjects see 3 characters, the character on eye glance and its previous and succeeding characters (Figure 4 (a)).
Alternately, when the center of eye glance isbetween characters, we suppose that subjects see
4 characters around the center of eye glance(Figure 4 (b)). Hereafter we call the area including characters glanced by subjects “fixationarea”.
After identifying the fixation area for eachfixation, we manually check if linguistic
boundaries exist in the fixation area. More con-cretely, we look for linguistic boundaries among
positions indicated by circles in Figure 4. Linguistic boundaries are defined by the entropy of
the 5-gram model: if the entropy at a position isgreater than a certain threshold T, we regard that
position as a linguistic boundary. Then the linguistic boundary ratio LBR, defined by the formula below, is calculated.

LBR=

No. of fixations such that at least one
linguistic boundary exists in the fixationarea
Total No. of fixations

LBR evaluates how likely fixations happen
around linguistic boundaries. If LBR is highenough, our hypothesis proposed in Section 2 isverified.

4 Experiment
4.1 Results of our Analysis
We found 367 backward saccades from the eye
tracking data of 20 subjects. The LBR is calculated for fixations at the start and at the end ofthese backward saccades. Results are summarized in Table 2. It shows LBR for several cases
classified according to the following aspects:
 Threshold of entropy (T)Since linguistic boundaries are defined according to entropy, the number of linguistic
boundaries in the text can be controlled by thethreshold T. In this experiment, we set thethreshold T to 2.5 and 3. Among 177 positions in
the text2, 53 (29.9%) and 36 (23.3%) positions
are regarded as linguistic boundaries when
T=2.5 and T=3.0, respectively.
 Start or End
Fixations are distinguished if they are at thestart or at the end of backward saccades. The
LBR for start fixations is shown in the left tables,
and the LBR for end fixations in the right tables.
 Groups of subjects
We separately calculate the LBR for fixationsof subjects in the 5 different groups described in
Subsection 3.1. In Table 2, the average and standard deviation (SD) of LBR for each group is
shown in each row. The same data for all subjects is also shown in the ‘All-ave’ row.
 Uncertain entropy position
As described in Subsection 3.2, the entropy atsome positions cannot be calculated due to data
2

Note that the text used in this experiment contains
178 characters.

Table 2 Results of Analysis

sparseness. If such uncertain entropy positions
are located in the fixation area, the judgment of
whether a linguistic boundary exists in the fixation area is also uncertain. So we calculated the
LBR when the entropy at all positions in the
fixation area can be calculated (‘Certain’ column
in Table 2), and the LBR for all fixations, including ones such that uncertain entropy positions exist in the fixation area (‘All’ column). In
the latter case, LBR is just an approximation.
Note that it is underestimated since uncertain
entropy positions may be real linguistic boundaries.
The expectation of the LBR is also shown inthe ‘E’ row in Table 2. It is defined as the proportion of points such that linguistic boundaries
exist in the neighborhood of all points in the text.
Here, points in the text mean both centers ofcharacters as indicated by the star in Figure 4 (a),and positions between characters as indicated by
the star in Figure 4 (b). That is, the expectationof the LBR represents a probability such thatwhen a point in the text represents randomly
chosen as the center of fixation, one or more
linguistic boundaries exist around that point.
4.2 Discussion
When T is set to 2.5, the averages of LBR for
all subjects are higher than the expectation of the
LBR for both start and end fixations of backward
saccades, as shown in Table 2. Furthermore, theaverages LBRs for most groups are also higher
than the expectation. This is almost true for start

fixations when T=3. These results indicate that
our hypothesis, namely Japanese learners tend to
start or end their backward saccades at linguistic
boundaries, is valid to some degree. On the other
hand, since the averages of the LBR are less thanthe expectation, our hypothesis is not valid forend fixations when T=3.
Next, we will discuss differences among 5groups of Japanese learners. Before the experiment, we expected that Japanese native speakers
might start or end their backward saccadesaround linguistic boundaries more often thanforeign students, since foreign students did nothave a good knowledge of Japanese. However,such a tendency is not observed. When T=2.5,
the LBR of group J (Japanese-Native-Speaker)
for start fixations is higher than that of othergroups, but this is not true for end fixations orwhen T is set equal to 3. Surprisingly, the LBR
of group NE (Non-Kanji-area-Europe) is relatively high for all cases. Although Europeansmight not be familiar with Kanji characters, theyoften start or end their backward saccadesaround linguistic boundaries. The above observations may suggest that the reading process ofJapanese learners is not strongly related to theirfamiliarity with Japanese. However, the size ofthe experiment is not sufficient in terms of boththe number of texts and the number of subjects.
It is necessary to conduct a larger experimentand to analyze the results further in order to reveal the reading process of Japanese learners.

5 Syntactic Structure and Entropy
Another question about backward saccades iswhether Japanese learners tend to start or end
their backward eye movements at deep positions
in syntactic structures of sentences. As a preliminary survey, we investigate the relationshipbetween depth of syntactic structures and theentropy of the N-gram model. First, we obtainedsyntactic structures of sentences in the text usedin our experiment with the Japanese CALL system Asunaro 3. For each character in sentences,
we obtained the entropy of the 5-gram model at
the position after the character and its depth in asyntactic tree, where depth is the distance fromthe root to the character. We obtained the statistics shown in Table 3.
Table 3 Depth in Structures and Entropy

The correlation between the depth and theaverage of the entropy is - 0.0941. This meansthat there is no relationship between the twovariables.

6 Conclusion
In this paper, we analyzed eye tracking data toinvestigate the reading process of Japaneselearners. First, we proposed the hypothesis thatlearners tend to start and end their backward
saccades around linguistic boundaries. The hypothesis is empirically verified by analyzing eye
tracking data when learners read a Japanese text
and by identifying linguistic boundaries according to the entropy of the N-gram model. The
results of our analysis indicate that our hypothesis is valid to some degree.
In future research, large-scale experiments are
required. We have already obtained eye-tracking
data for 3 other texts with the same subjects. The
analysis of these 3 texts will be performed
shortly. A detailed comparison between nation3

http://hinoki.ryu.titech.ac.jp/

alities and languages of Japanese learners will
also be made. Furthermore, we plan to empirically investigate the relation between backward
saccades and the depth in syntactic structures.

References[1] Ford, Bresnan, and Kaplan. A competence-based
theory of syntactic closure. In Joan Bresnan, editor,
The Mental Representation of Grammatical Relations,
MIT Press, 1982.
[2] Naoaki Osaka. Size of saccade and fixation duration of eye movements during reading. Psychophysics of Japanese text processing, Journal of the OpticalSociety of America, A (9): 5-13, 1992.
[3] Ryoji Osaka, Yukio Nakazawa and Kazuo Koga.
Experimental psychology of eye movement. Nagoya
University press, Tokyo, Japan, 1993.
[4] Jun Shigematsu and Tsutomu Konosu. The research on the Reading Processes by Non-Native
Speakers using on Eye-Camera. Spring Proceedingsof the KNG: 31-42, 1993. (in Japanese)[5] Akemi Tera, Tatsuya Kitamura and Koichiro
Ochimizu. Japanese Reading Support System “dictlinker”. Autumn Proceedings of the KNG: 43-48,
1996. (in Japanese)[6] Akemi Tera. Extensive Reading Support System
for Learning Kanji. Meiji Shoin, 16(6): 101-108,
1997. (in Japanese)[7] Akemi Tera, Tatsuya Kitamura, Koichiro Ochimizu, Tomoko Graham and Ann Lavin. JapaneseReading Support System (DL) Evaluation - Resultsfrom MIT User Survey. JLEM, 4(1): 26-27, 1997. (inJapanese)[8] Akemi Tera, Tatsuya Kitamura and Manabu
Okumura. The Verification of Japanese Reading
Support System "DL" -The Research of The Processof Japanese learner's reading. JLEM, 6(1): 20-21,
1999. (in Japanese)[9] Christopher D. Manning, Heinrich Schutze.
Foundations of Statistical Natural Language Processing. The MIT Press, 1999.
[10] Akemi Tera, Hajime Motizuki and Akira Shimazu. An analysis of the words that Japanese Learners need Information during Reading. Autumn Proceedings of the 27th JSISE Conference: 259-260,
2002. (in Japanese)[11] Chunichi Newspaper. Chunichi Newspaper
Company, 2004.
[12] Akemi Tera and Kozo Sugiyama. Eye-Tracking
Analyses of Japanese Reading Processes (IV) - Forward and Backward Jumping of Eye Movements-.
Proceedings of the IEICE, TL2005-35-48: 43-48,
2006. (in Japanese)[13] Akemi Tera and Kozo Sugiyama. Eye-Tracking
Analyses of Japanese Reading Processes. Proceedings of the KICSS: 108-114, 2006.