Eye can tell you how Japanesemorphographic words are recognized― A lexical decision with eye-tracking study ―Koji Miwa1, Gary Libben2, Ton Dijkstra3, & Harald Baayen1
1 University of Alberta, Canada
2 University of Calgary, Canada
3 Radboud Univeristy,

DCC language group meeting
September 28, 2009

Reading a Japanese Text In Japanese, there are three types of orthography
1. Morphographic kanji（阿蘭陀）

2. Moraic hiragana（おらんだ）

3. Moraic katakana（オランダ）(4. Alphabetic Romaji)（Nezarando) Roughly 70% of Japanese words are two-kanji-character words.

2Reading a Japanese Text

3c.f. Kajii,Nazir, & Osaka (2001)

Japanese Kanji Logography
 The majority of Japanese“simplex” words consist ofmorphographic characters.
 Majority of characters further
consist of a semantic radical
and a phonetic radical.
 1,945 basic characters in total.
 How are two-characterwords recognized ?

4

Purely orthographic morpheme ?

Recognizing an Unfamiliar Word No need to know the whole No need to know its pronunciation‘cold’‘fish’

Studies on Compound word recognition
 How efficiently are complex words represented and processed ?
(Libben, 2005, OUP)
① Maximizing computational efficiency? => Pre-sorting is necessary② Maximizing storage efficiency? => No advantage for HF words③ Maximizing opportunities? => Lexicalize words but still process everything①②[[boat][house]][boat][boat][house][houseboat]

 Sub-lexical vs. Supra-lexical③[house]boat
house[[house][boat]]
6

Single-route vs. Dual-route vs. Multi-route What architecture best explains the complexity in Japanese ?

Whole Two-Character Words as
Representational Units Morton, Sasanuma, Patterson, & Sakuma (1992)- Full word forms are recognition units- Only identical full forms cause priming effect(in a long-lag priming experiment)

7

Characters as Representational Units Hirose (1992)’s partial constituent priming study- Left characters are important- Two-character words with the same left character are tied
together in the mind
 Tamoka & Hatsuzuka (1995)’s lexical decision- Right characters are important- This is because of the left-to-right reading sequence8

Characters as Representational Units
Tamaoka & Hatsuzuka (1998)

 Both the whole word unit andthe character unit has theirsemantic representation.
 No radical-level
representation
 The order of processing isunder-specified

 Are radicals important ?

9Radicals as Representational Units
Saito (1997)

 There is a radical-level Only the whole word unit has
a semantic representation
 Characters are notconsidered as morphemes A single-route model Do characters behave likemorphemes?
 Are radicals processed first?

10

Radicals as Representational Units
Taft & Zhu (1997)

 Both the whole word unit andthe character unit has theirsemantic representation.
 Semantic radicals are notconsidered as orthographicmorphemes.
 A single-route model Do radicals behave likemorphemes?
 Are radicals processed first?

11What We Want to Study Language-general issues
1. Locus of whole word frequency effect (c.f. Kuperman et al., 2009)
2. Relative importance of the left and the right constituents in compounds
3. Order of activating the constituents and the whole word representations
4. Role of attention (what do eye fixations reflect?) Language-specific issues
1. Whether semantic radicals contribute in the larger context of default
two-character word recognition (no evidence so far).
2. Status of characters (morphemes? or orthographic components?)
3. Status of radicals (orthographic morphemes? or orthographic components?)
12

Lexical Decision with Eye-Tracking
 Participants- 21 native Japanese readers in Canada Materials- 708 two-character words (=> 555)
- 708 nonwords (various types)
 Procedure- 472 words in one session * 3 sessions- Fixation point manipulation(left, central, right)

13

20 Lexical Properties We ConsideredResidualization to Handle Multicollinearity
 Complex correlational structure is inherent among lexical distributional
properties.

15

RT Analysis
 A model that stores frequent complex word forms forcomputational efficiency=> there should be non-linear/ catastrophic whole word
frequency effect A model that always compute complex words from thescratch without storing complex word forms=> Complex words’ frequency is not importantRT Analysis

17

RT Analysis

-3.0

-2.0

-1.0

5

7

9750

7

9550

600

650

700

5

RT (ms)

700
650
600

RT (ms)114.8

LeftKanjiTokenFreq

-8.3

13.6
11.9
10.91011-3-1

1 2

Whole Word Freq

Left Kanji Neighbour(e) - Character x Task(f) - Character(g) - Radical x Feature(h) - Individual x Task

1012

RightKanjiTokenFreq-202

Right Kanji AoA (Resid)

1234

Right Kanji Radical Combinability

700
650

Right
Left

Fixation

600

Central

550

700
600
550

-45.7

650

600
550

650

RT (ms)700

8

RT (ms)PreviousTrialCorrect

650
600
550

6-3.2
-1.3
-0.4
1.1

RT (ms)

750

750

750
700

Correct

750

Whole Word Freq

RightKanjiRadicalStrokesResid

Previous RTIncorrect

RT (ms)(d) - Character

750(c) - Whole Word x Character

550

550

550

2.3-0.5-3.2

LeftKanjiStrokesResid

750
700
650

11.9600

650

RT (ms)

750700(b) - Whole Word x Feature

600

RT (ms)(a) - Task

01234

Length of Stay in Canada

18What is “Length of stay in Canada” ?

Reading
Proficiency
In ENGL

Daily
Exposureto JapaneseCharacter
Reading
Ability

Vocabulary
Size

Length ofStay InCanada

19Interim Summary: RT Analysis

 Character frequency still exerts significant effect when AoA is
considered together.
 Both the left character and the right character are important, andthey are qualitatively different in their role
 The right character’s semantic radical effect is manifested in RTs
 Semantic radicals may not be fully morphemic.

20

Eye-Movements during Lexical Decision

 Eyes try to optimize word recognition process by moving to optimalpositions.
 Sentence reading vs. Isolated word reading

21

1st gaze duration for trials with
2 fixations

510-50

5400
250

300

350

Left200

EyePosition

Right150

200

250

300

1st Gaze Durations (ms)350

400(d) - Character x Task

150200EyePosition

Right

1st Gaze Durations (ms)350
250300Left

150

EyePosition0

1st Gaze Durations (ms)

350
300
250
200

Right150

101234681012(e) - Character x Task(f) - Character(g) - Whole Word(h) - Whole Word

-3-1 0

1

2

Left Kanji Neighbour (Resid)-3-1 0

1

2

Right Kanji Neighbour (Resid)

350
300
250
200

1st Gaze Durations (ms)

350
300
250

22150

200

1st Gaze Durations (ms)

350
300
250
200

150

150

200

EyePosition

Right

1st Gaze Durations (ms)

350
250

300

Left

400

Left Kanji Token Freq

400

Left Kanji Radical Combinability

400

Right Kanji Strokes (Resid)

400

Left Kanji Strokes (Resid)150

1st Gaze Durations (ms)

Left

-5

1st Gaze Durations (ms)(c) - Radical

400(b) - Feature x Task

400(a) - Feature x Task

-3-1

1

3

Google Doc Freq (Resid)

57

911

Whole Word Freq

2nd gaze duration for trials with2 fixations
Left(d) - Character
400(c) - Radical
400(b) - Feature x Task
400

400

(a) - Feature x Task

05051234(f) - Whole Word(g) - Whole Word01

2

Right Kanji Neighbour (Resid)

350
300

81012

Right Kanji Token Freq

350
300
250
200

2nd Gaze Durations (ms)

350
300
250

23150

200
150

EyePosition-1

2nd Gaze Durations (ms)

350
300
250
200
-36400(e) - Character
400

Right Kanji Radical Combinability

400

Right Kanji Strokes (Resid)

Right

250
150

10

Left Kanji Strokes (Resid)

Left

200

2nd Gaze Durations (ms)

350
300
250
200
150

EyePosition-5

2nd Gaze Durations (ms)

350
300
250
200

10150

2nd Gaze Durations (ms)Right150

EyePosition

200-5

2nd Gaze Durations (ms)

350
300
250

Right150

2nd Gaze Durations (ms)Left57911Whole Word Freq

-3-113

Google Doc Freq (Resid)

1st gaze duration for trials with
3 or more fixations
400(d) - Radical x Individual

400(c) - Radical x Individual

400(b) - Feature x Task

400(a) - Feature x Task

-505-5510-2-101(e) - Character x Task(f) - Character x Task(g) - Whole Word

681012

Left Kanji Token Freq

3

LengthOfStayCanada

350
300250200

24

LeftKanjiRadicalCombinability

681012Right Kanji Token Freq

350
300
25020024150

EyePosition

Right1st Gaze Durations (ms)350
300200250Left

150

EyePosition

Right

1st Gaze Durations (ms)

350
200

250300Left

1400

Left Kanji Radical Token Freq (Resid)

400

Right Kanji Strokes (Resid)

400

Left Kanji Strokes (Resid)

0
2.1
3.2
4.1
4.3

150

LengthOfStayCanada

200

250

300

3.2
4.1
4.3

1st Gaze Durations (ms)

350
2.1

150

EyePosition0

1st Gaze Durations (ms)

350
300
250
200

Right10150

1st Gaze Durations (ms)

Left

150

EyePosition

Right

1st Gaze Durations (ms)

350
200

250300Left

150

1st Gaze Durations (ms)

0-3-1

1

3

Google Doc Freq (Resid)Summary All representations that can be activated can be activated(maximization of opportunity principle)
 Whole word frequency effect arises early before the whole word
is scanned, but nature of this effect is yet to be investigated The relative importance of the left and the right constituentsshould not be just understood quantitatively. There is qualitative
difference in their role (c.f. Gagne & Shoven, 1997, CARIN theory) Semantic radicals are processed but to a slight extent
 It is preferable to process from left to right What we are fixating ≠ what we are processing (the eye-mind
assumption does not hold perfectly).
 Attending what is supposed to be processed at an appropriate
point in time is, however, important.

25

Now, What is the Architecture !!
 The eye-tracking results do
not fully support the current
IA models of morphographic
word recognition in Chineseand Japanese.

26

Now, What is the Architecture !!
 The radical-level is skipped,
and radicals are activatedthrough characters.
 Semantic radicals are activatedto maximize opportunities, butthey are not always useful Characters and semantic
radicals have their own semantic
representation.

 This IA architecture still needs
to explain the early whole word
frequency effect and resolve
the issue of a feature template.

27Issues that can make my life
even more enjoyable in the future What happened during the 1st gaze duration ?
 What is qualitative nature of the early whole word frequency?
(perceptual ? or conceptual ?) Role of other properties of compounds ? (e.g. transparency)
 Word / Character identification speed
 What about other tasks ? (e.g. naming, PDM)
 Chinese vs. Japanese ?
 What about other groups of individuals ?
…. and many more28

Language Effects are Small PrevRT
WholeWordFreq
LengthOfStayCanada
Trial
GoogleDocFreqResid
LeftKanjiTokenFreq
RightKanjiTokenFreq
LeftKanjiStrokesResid
RightKanjiHomophones
LeftKanjiRadicalStrokesResid
RightKanjiAoAResid
LeftKanjiRadicalCombinability
RightKanjiStrokesResid
LeftKanjiAoAResid
RightKanjiRadicalCombinability
RightKanjiNeighbourRightPositionResid
RightKanjiRadicalTokenFreq
LeftKanjiRadicalTokenFreqResid
LeftKanjiHomophones
RightKanjiRadicalStrokesResid
LeftKanjiNeighbourLeftPositionResid
LeftKanjiOnKun
RightKanjiOnKun

Random forests(Strobl et al, 2007, 2008)

290.00

0.01

0.02

0.03

0.04

'Mean decrease in accuracy' importance scores

Thank You
Correspondence
Koji Miwa

Email: kmiwa@ualberta.ca

This research was supported by a Major Collaborative Research Initiative Grant(#412-2001-1009) from the Social Sciences and Humanities Research Council ofCanada to Gary Libben (Director), Gonia Jarema, Eva Kehayia, Bruce Derwing, andLori Buchanan (Co-investigators).

30

Density Plots for Different Fixation
Positions in Sequence

31

32

33

2 GAZE DURATION FOR TRIALS WITH 3 FIXATIONSND34

3 GAZE DURATION FOR TRIALS WITH 3 FIXATIONS
RD35