Orthographic support for passing
the reading hurdle in Japanese

A thesis presentedbyLars Yencken
toe Department of Computer Science and Software Engineeringin partial fulﬁllment of the requirements
for the degree ofDoctor of Philosophy

University of Melbourne
Melbourne, Australia
April 2010©2010 - Lars Yencken
All rights reserved.

esis advisor(s)Author

Timothy Baldwin

Lars Yencken

Orthographic support for passing
the reading hurdle in Japanese

Abstract
Learning a second language is, for the most part, a day-in day-out struggle against themountain of new vocabulary a learner must acquire. Furthermore, since the number of newwords to learn is so great, learners must acquire them autonomously. Evidence suggeststhat for languages with writing systems, native-like vocabulary sizes are only developed
through reading widely, and that reading is only fruitful once learners have acquired the core
vocabulary required for it to become smooth. Learners of Japanese have an especially highbarrier in the form of the Japanese writing system, in particular its use of kanji characters.
Recent work on dictionary accessibility has focused on compensating for learner errors inpronouncing unknown words, however much diﬃculty remains.
is thesis uses the rich visual nature of the Japanese orthography to support the studyof vocabulary in several ways. Firstly, it proposes a range of kanji similarity measures andevaluates them over several new data sets, ﬁnding that the stroke edit distance and tree edit
distance metrics best approximate human judgements. Secondly, it uses stroke edit distanceconstruct a model of kanji misrecognition, which we use as the basis for a new form ofkanji search by similarity. Analysing query logs, we ﬁnd that this new form of search was
rapidly adopted by users, indicating its utility. We ﬁnally combine kanji confusion andpronunciation models into a new adaptive testing platform, Kanji Tester, modelled after
aspects of the Japanese Language Proﬁciency Test. As the user tests themselves, the system
adapts to their error patterns and uses this information to make future tests more diﬃcult.
Investigating logs of use, we ﬁnd a weak positive correlation between ability estimates andtime the system has been used. Furthermore, our adaptive models generated questions
which were signiﬁcantly more diﬃcult than their control counterparts.

iv

Abstract
Overall, these contributions make a concerted eﬀort to improve tools for learner self-

study, so that learners can successfully overcome the reading hurdle and propel themselves
towards greater proﬁciency. e data collected from these tools also forms a useful basis forfurther study of learner error and vocabulary development.

Abstractv

is to certify thati. the thesis comprises only my original work towards the PhD
ii. due acknowledgement has been made in the text to all other material usediii. the thesis is less than 100,000 words in length, exclusive of tables, maps, bibliographies
and appendicies

Lars Yencken

Citations to Previously Published Work

Large portions of Chapter 4 have appeared in the following papers:
Yencken, Lars and Timothy Baldwin. 2008. Measuring and predicting orthographic associations: modelling the similarity of Japanese kanji.
In Proceedings of the 22nd International Conference on Computational Linguistics,
1041–1048, Manchester, UK.
Yencken, Lars and Timothy Baldwin. 2006. Modelling the orthographic neighbourhood for Japanese Kanji. In Proceedings of the 21st International Conference on Computer Processing of Oriental Languages, 321-332, Sentosa, Singapore.
Similarly, large portions of Chapter 5 have appeared in:
Yencken, Lars and Timothy Baldwin. 2008. Orthographic similarity
search for dictionary lookup of Japanese words. In Proceedings of the 18th European Conference on Artiﬁcial Intelligence, 343–347, Patras, Greece.
Yencken, Lars and Timothy Baldwin. 2005. Eﬃcient graphemephoneme alignment for Japanese. In Proceedings of the Australasian Language
Technology Workshop 2005, 143–151, Sydney, Australia.

Acknowledgmentsere are many people who have supported me through this long adventure. I’d liketo begin by thanking my parents for their persistent and dogged encouragement, my wife
Lauren for her love and understanding, and my cat Nero for his quiet, enduring company
along the way.
When I ﬁrst began this work, Slaven Bilac visited our lab in Melbourne and handed
over both source code and control of the FOKS system so that I could use it as a test bed for
new dictionary research. I would like to thank him for his kindness and generosity, without
which much of this work would not have been possible.
In late 2006 I spent 6 months with the Tanaka Lab at the University of Tokyo. I would
like to thank Kumiko Tanaka-Ishii for kindly supporting my visit, and for expanding myperspective on many aspects of Japanese life, language and culture. My warmest thanks go
also to my friend and colleague Zhihui Jin for our many long discussions during this period,and for his fresh insights as a Chinese learner of Japanese.
Finally, my most sincere and wholehearted thanks go to my long-suﬀering supervisor
Tim Baldwin, for his sharp insight, stoic patience, constant warmth and all round good
humour.
ank you.

Dedicated to Lauren, my wife and partner in all things.

Contents
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

xii
xvi

1

Introduction
1.1 Aim and contributions . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.3 esis structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1
5
6
6

2

e Japanese writing system
2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Input and lookup of Japanese . . . . . . . . . . . . . . . . . . . . . . . .
2.3 Word formation in Japanese . . . . . . . . . . . . . . . . . . . . . . . .

10
10
14
18

3

Learning Japanese vocabulary
3.1 Autonomous vocabulary acquisition . . . . . . . . . . . . . . . . . . . .
e beginner’s paradox . . . . . . . . . . . . . . . . . . . . . . . . . . .
Knowing a word . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Ordering strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Acquisition strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Self-study tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Structure of the mental lexicon . . . . . . . . . . . . . . . . . . . . . . .
Visual word recognition . . . . . . . . . . . . . . . . . . . . . . . . . . .
General lexical relationships . . . . . . . . . . . . . . . . . . . . . . . .
Graphemic neighbourhoods: errors and associations . . . . . . . . . . . .
3.3 Dictionary lookup of Japanese . . . . . . . . . . . . . . . . . . . . . . .
Lookup for producers . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Lookup for receivers . . . . . . . . . . . . . . . . . . . . . . . . . . . .
FOKS dictionary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4 Second language testing . . . . . . . . . . . . . . . . . . . . . . . . . . .
e role of testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Testing theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Vocabulary testing and drilling . . . . . . . . . . . . . . . . . . . . . . .

22
23
23
26
29
30
36
41
42
46
49
52
52
53
56
57
57
59
61

ix

x

Contents
3.5

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

63

4Orthographic similarity
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Distance within alphabetic orthographies . . . . . . . . . . . . . . . . .
4.2 Distance models: a ﬁrst approach . . . . . . . . . . . . . . . . . . . . . .
Form of models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Bag of radicals model . . . . . . . . . . . . . . . . . . . . . . . . . . . .
L1 norm across rendered images . . . . . . . . . . . . . . . . . . . . . .
4.3 Similarity experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Experiment outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Control pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Evaluating similarity models . . . . . . . . . . . . . . . . . . . . . . . .
Model evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4 Distance models: a second approach . . . . . . . . . . . . . . . . . . . .
Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

64
64
64
64
66
66
67
68
70
70
72
72
74
75
77
78
82
89
90

5

Extending the dictionary
5.1 FOKS: the baseline . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Intelligent lookup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Error models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
FOKS rebuilt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2 Grapheme-phoneme alignment . . . . . . . . . . . . . . . . . . . . . . .
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Existing algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Modiﬁed algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Improved alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3 Usability enhancements . . . . . . . . . . . . . . . . . . . . . . . . . . .
Place names and homographs . . . . . . . . . . . . . . . . . . . . . . . .
Sense separation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Query and reading explanation . . . . . . . . . . . . . . . . . . . . . . .
5.4 Intelligent grapheme search . . . . . . . . . . . . . . . . . . . . . . . . .
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Overall model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

92
93
93
95
97
97
99
99
99
101
105
108
112
113
113
115
115
118
118
119

Contents

xi

5.5
5.6

Confusion model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
Log analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

6

Testing and drilling
6.1 From search to testing . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Testing as an application space . . . . . . . . . . . . . . . . . . . . . . .
e ideal test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.2 Japanese Language Proﬁciency Test . . . . . . . . . . . . . . . . . . . .
Overview of the JLPT . . . . . . . . . . . . . . . . . . . . . . . . . . .
Question types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Potential criticisms of the JLPT . . . . . . . . . . . . . . . . . . . . . .
6.3 Kanji Tester . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
User interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
User modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Generating questions . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Update algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
User demographics . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
User ability and improvement . . . . . . . . . . . . . . . . . . . . . . . .
Power users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Measuring adaptiveness . . . . . . . . . . . . . . . . . . . . . . . . . . .
Error models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

128
129
129
130
132
132
134
137
139
139
144
148
151
155
155
157
163
166
169
171
173

7

Conclusions
7.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Graphemic similarity . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Dictionary search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Vocabulary testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.2 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Lexical relationships . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Dictionary search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Proﬁciency testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

175
175
176
177
178
178
178
179
181
182

Bibliography

183

Index

194

List of Figures
2.1

2.2

2.3

3.1
3.2
3.3

3.4
3.5

3.6
3.7
3.8

Exposing the compositional nature of kanji. 漁 asa(ru) “ﬁshing” is made bycombining 水 mizu “water” (via short form 氵) and 魚 sakana “ﬁsh”. Further
decomposition of 魚is possible, however its subcomponents do not have asemantic relationship with its whole; this is because 魚is a pictograph (i.e.
it is derived from a picture of a ﬁsh rather than from these components).
e etymology for the character 歳 sai “year, age” from oracle characters tomodern day usage in Chinese and Japanese, and the cognate relationshipbetween the two usages. Cognates are from Goh et al. (2005), and early
character forms are from Sears (2008). . . . . . . . . . . . . . . . . . . .
An example of IME use to type in Japanese. e pronunciation is ﬁrst entered, then matching word forms are displayed to choose from. e example
reads カレーを食べよう karē o tabeyō “let’s eat curry” . . . . . . . . . . .
Nuttall’s (1996) cycles of frustration and growth in L2 reading. . . . . . .
e basic study mode of Quizlet, shown part way through study of the JLPT
4 kanji set. e JLPT 4 kanji list was entered by another user. . . . . . . .
e “Scatter” game in Quizlet, where users must drag and drop matchingelements of word-gloss pairs onto each other to eliminate them. e user
is scored by their time taken, which is compared to their previous best. . .
e card system proposed by Leitner for spaced repetition. From Mondria(2007). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Rikai used to aid reading of a news article from Asahi Shimbun. When themouse is hovered over the word 商会, its gloss “ﬁrm; company” is displayed,
as well as an individual gloss for each kanji in the word. . . . . . . . . . .
e multilevel interactive-activation model for alphabetic orthographies.
From McClelland and Rumelhart (1981). . . . . . . . . . . . . . . . . .
e Companion Activation Model (CAM). From Saito et al. (1998). . . .
Lemma unit connections in Joyce’s (2002) multilevel interactive-activation
model for Japanese. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
xii

12

15

16
26
37

38
39

40
43
44
45

List of Figures
Interlingual relationships based on near- homophony, homography andsynonymy. e ﬁrst example is English-French; the second and third areJapanese-Chinese. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.10 Kanji lookup methods to ﬁnd 明 aka “bright”. Steps beginning with “IDENTIFY” or “COUNT” are potential error sites for learners. Steps beginningwith “LOOKUP” involve use of an index, and subsequent visual scan forthe desired item. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

xiii

3.9

Drawing rough equivalence between the linguistic units used in English
and Japanese. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 Kanji decomposed using radkﬁle into their naive radical-member feature set
representations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3 Example stimulus pair for the similarity experiment. is pair contains ashared radical on the left. . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4 Groups of control pairs used, with an example for each. Parts of readings inbrackets indicate okurigana, necessary suﬃxes before the given kanji forms
a word. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.5 Participant responses grouped by language background, measured over thecontrol set stimulus. On the left we give the mean response for each group;on the right, the mean pairwise rank correlation between raters of the samegroup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.6 Rank correlation of pixel and radical models against raters in given participant groups. Mean and median raters provided as reference scores. . . . .
4.7 Rank correlation of pixel and radical models against raters in across bandsof kanji knowledge. Each band contains raters whose number of known
kanji falls within that band’s range. . . . . . . . . . . . . . . . . . . . . .
4.8 Histograms of scaled responses across all experimental stimulus pairs, takenfrom mean rating, pixel and bag of radical models. Responses were scaled
into the range [0, 1]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.9 A summary of our kanji distance metrics . . . . . . . . . . . . . . . . . .
4.10 Mean value of Spearman’s rank correlation ρ, calculated over each ratergroup for each metric. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.11 An example White Rabbit Press ﬂashcard for the kanji 晩 baN “evening”.
Note the two large characters on the right provided due to visual similarity
with the main kanji. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.12 Example stimulus from the distractor pool experiment. For each kanji on
the left-hand side, participants could mark one or more of the kanji on theright as potentially confusable visual neighbours. . . . . . . . . . . . . . .

48

54

4.1

5.1
5.2

FOKS search results for the reading kazeja. e ﬁrst result 風邪 kaze “common cold” is the desired word. . . . . . . . . . . . . . . . . . . . . . . .
e architecture of the original FOKS interface. . . . . . . . . . . . . . .

65
67
71

73

74
75

76

76
79
83

84

88
94
95

xiv

List of Figures
e dictionary entry for 感謝する kaNshasuru “to give thanks, be thankful”,
with two of its potential alignments shown. . . . . . . . . . . . . . . . .
5.4 e TF-IDF based alignment algorithm . . . . . . . . . . . . . . . . . .
5.5 Disambiguation using the reading model . . . . . . . . . . . . . . . . . .
5.6 Examples of incorrect alignment in the combined task . . . . . . . . . . .
5.7 e distribution of place-name homography in Japanese. e vertical axis
shows how many place names had at least n homographs, out of the 79139
unique place names encountered. . . . . . . . . . . . . . . . . . . . . . .
5.8 e old and new translation pages shown for the compound 下手. . . . .
5.9 FOKS explaining the relationship between the query tokyoand the word
東京 Tōkyō. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.10 e potential accessibility improvement of kanji similarity search . . . . .
5.11 Histogram of log query/target likelihood ratio for all corrected grapheme
queries, taken on a per-kanji basis. . . . . . . . . . . . . . . . . . . . . .
5.3

e various types of Japanese proﬁciency testing as they vary in scope andavailability. e dotted line represents the current state-of-the-art. . . . .
6.2 An example results card from the JLPT 2007 tests. . . . . . . . . . . . .
6.3 Easily testable aspects of word knowledge for words containing kanji, andwords without kanji. A dotted line indicates a more tenuous link. . . . . .
6.4 Example question: the learner must select the correct reading for each kanji
word in context. Taken from the JLPT level 3 example questions, Writing
and Vocabulary section. . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.5 Example question: the learner must select the correct kanji form for words
given by their reading. Taken from the JLPT level 3 example questions,
Writing and Vocabulary section. . . . . . . . . . . . . . . . . . . . . . .
6.6 Creating a proﬁle for Kanji Tester. e user enters a ﬁrst language, andoptionally any second languages they have studied. . . . . . . . . . . . . .
6.7 A sample test generated by Kanji Tester for JLPT 3. . . . . . . . . . . . .
6.8 An example question measuring the surface-to-reading link for the word
用意 yōi “preparation”. . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.9 e feedback given to a user immediately upon completing a test. . . . . .
6.10 Examples of user and error models of varying granularity considered foruse by Kanji Tester. On the left, user models can be maintained globally(treating all users identically), in aggregate by some common attribute (e.g.
ﬁrst language), or individually. Similarly error prevalence can be modelled
globally, by broad error type, by kanji or by word. e actual conﬁgurationof Kanji Tester as reported models users as individuals and errors at the kanji
level. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.11 e high-level architecture of Kanji Tester. Dotted lines indicate areas designed for easy extension. . . . . . . . . . . . . . . . . . . . . . . . . . .
6.12 e algorithm used by Kanji Tester to generate each test question. . . . .

101
102
107
112

114
116
117
121
125

6.1

131
134
135

136

136
140
141
142
143

146
149
150

List of Figures
6.13 Number of users (reverse cumulative) set against number of tests. . . . . .
6.14 e proportion of times between tests taken sequentially by a user, expressedas the proportion of such times of magnitude > x hours (reverse cumulative
histogram). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.15 Mean score per user as a histogram. . . . . . . . . . . . . . . . . . . . .
6.16 Ability, as measured by mean score for each user, plotted against usage, as
measured by the total number of tests each user took. . . . . . . . . . . .
6.17 Ability, as measured by test score on the nth test, plotted against study time,
as measured by number of tests taken n, for all users. . . . . . . . . . . .
6.18 User scores plotted against time in days since their ﬁrst test. . . . . . . . .
6.19 e diﬀerence in scores between the ﬁrst time a stimulus was seen and thelast time it was seen, expressed as a histogram. . . . . . . . . . . . . . . .
6.20 State transition diagram for known and unknown items. In this context“known” mean the user answered the most recent question about the item
correctly. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.21 Accuracy and exposure for words and kanji by the top-5 power users. Eachgraph shows the total number of unique items tested after the nth test, andthe number of items responded to correctly upon last encounter. . . . . .
6.22 Error rates compared for simple and adaptive plugins. . . . . . . . . . . .

xv
156

157
158
159
159
160
161

162

165
168

List of Tables
3.1
3.2
3.3
3.4

4.1
4.2
4.3
4.4
4.5
4.6
5.1
5.2
5.3
5.4
5.5
5.6
5.7

Aspects of word knowledge, marked “R” for receptive knowledge and “P”
for productive knowledge. From Nation (2001:27). . . . . . . . . . . . .
Our proposed aspects of kanji knowledge, in complement to aspects of word
knowledge shown in Table 3.1. . . . . . . . . . . . . . . . . . . . . . . .
Types of lexical relationships available to the ideal bilingual speaker. . . .
Four types of question used by Laufer and Goldstein (2004) in CATSS,
each indicating the task the learner must perform. For each condition ⊕indicates increased diﬃculty, ⊖indicates decreased diﬃculty. . . . . . . . .
Groups of characters which share an identical feature-set representation according to radkﬁle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Accuracy at detecting which of two pairs (ﬂashcard vs. random) has highsimilarity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
e mean average precision (MAP), and mean precision at N ∈ {1, 5, 10}
over the ﬂashcard data . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Pairwise signiﬁcance ﬁgures for MAP scores over the ﬂashcard data, shownas p values from a two-tailed Student’s t-test . . . . . . . . . . . . . . . .
e mean average precision (MAP), and mean precision at N ∈ {1, 5, 10}
over the pooled distractor data . . . . . . . . . . . . . . . . . . . . . . .
Pairwise signiﬁcance ﬁgures for MAP scores over the pooled distractor data,
shown as p values from a two-tailed Student’s t-test . . . . . . . . . . . .
Alignment accuracy across models. . . . . . . . . . . . . . . . . . . . . .
Alignment execution time across models in minutes and seconds. . . . . .
Okurigana detection accuracy across models . . . . . . . . . . . . . . . .
Best model accuracy for the combined task . . . . . . . . . . . . . . . . .
FOKS usage as reported by Google Analytics in the period from October2007 to March 2009, broken down by browser reported language. . . . . .
Number of FOKS searches logged between October 2007 and March 2009,
arranged by search type. . . . . . . . . . . . . . . . . . . . . . . . . . . .
Automatically determined error types for intelligent queries by reading. . .
xvi

27
28
46

61
68
85
86
86
89
89
109
109
110
111
122
123
124

List of Tables
6.1

6.2
6.3
6.4
6.5
6.6
6.7
6.8

e four levels of the JLPT, with approximate numbers of kanji and words
which learners should know when testing at each level, and estimated hoursof study required. From Japan Foundation (2009). . . . . . . . . . . . . .
Sections of a JLPT exam and their relative weighting. From Japan Foundation (2009). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Distribution of active users by ﬁrst language. . . . . . . . . . . . . . . . .
Basic user statistics for the 5 users with the most responses. . . . . . . . .
Plugin name, type and number of questions generated. . . . . . . . . . .
Mean number of exposures across all users by item type. e Kanji Combined type counts words containing kanji as exposures for those kanji. . .
Error types for adaptive reading questions. . . . . . . . . . . . . . . . . .
Top 8 words or kanji by number of grapheme substitution errors. . . . . .

xvii

133
133
156
163
166
167
170
170

Chapter 1
Introduction
Words are the basic building blocks of language, and acquiring a large enough vocabulary to both understand others and be understood is the dominant task in second language
learning. For learners of Japanese, learning words means learning the writing system used to
encode them, in particular the kanji characters used to write them. Kanji are complex characters which embed meaning in a hierarchical, two-dimensional manner, in contrast to thelinear, phonetic manner of most alphabetic scripts. is thesis centres around kanji, modelling how they may be confused by learners and in turn demonstrating how these confusion
models can be used to improve dictionary lookup and automated test generation. roughthis contribution, it aims to help learners in their self-study of Japanese vocabulary, so thatthey may better achieve their proﬁciency goals.
To make this more concrete, let us consider a common scenario of a hypothetical learner,
Jane. Jane studies Japanese as a second language, but struggles constantly with the large
number of words she has to learn. Since there are so many, most are only glossed over
brieﬂy in class and almost all her vocabulary study is done at home in her own time. Really,
she would love to be reading in Japanese, but when she has tried even simple texts she ﬁnds
she has to look up every second word in the dictionary.
One day she is trying to read a Japanese article, and comes across a new word, 養い. esecond half い is phonetic, pronounced i, but she has no idea about the ﬁrst half. Nothing
about the character 養 indicates its pronunciation in this context, and without the pronunciation she cannot type the word into an online dictionary. Even if such a dictionary12

Chapter 1: Introduction

supported wildcards, i is a very common word ending, and she would never ﬁnd her word.
Jane realises that she must do things the hard way, and look it up in her paper dictionary.
In order to look the word up, she sets about trying to ﬁnd the ﬁrst character in a traditional
kanji dictionary. To do this, she has to identify the indexing radical, the component of thecharacter traditionally used in dictionary indexes, usually the left-most or top-most component. Often components within characters are clearly delineated by white space, but inthis character they are not. She guesses that the top half is probably the indexing radical,
counts that it has 6 strokes, and tries to ﬁnd it amongst the list of 6-stroke radicals in the
dictionary. However, the nearest match she can ﬁnd is 羊, which isn’t quite what she waslooking for because its main vertical stroke is too long at the bottom. She nonetheless continues just in case, going to the page number indicated for kanji containing 羊, counts thatthe rest of 養 would have 9 remaining strokes, and looks in the 9-stroke section for a match,without success.
At this point, with some frustration, she tries to work out what went wrong, rechecking
stroke counts and wondering how else she could ﬁnd this character in a dictionary. Morefrustrating still, Jane can recognise at least the bottom half of the kanji. It looks just like 食from 食事 shokuji “meal”. en she recalls a recent online dictionary improvement whichallows the user to search for words by visual similarity. She loads up its web site, and decides to put in the query 食い. e search results yield two words based on kanji which look
similar to 食, and the second of these is the word she was looking for. She clicks “Translate”, and ﬁnally reaches the information she needs. Her word is 養い yashinai, meaning“nutrition”.
e broad problem Jane faces is really one of insuﬃcient vocabulary, and it is common
to all language leaners. It severely impedes early attempts at reading, since the time ittakes to look up the many unknown words encountered makes comprehension diﬃcult, andenjoyment unlikely. However, some of the additional diﬃculties faced by Jane are speciﬁc
to learners of Japanese, especially from an English or Western language background, andrelate to the problem of unknown kanji characters. Once they have acquired a small base oforal vocabulary, learners often study a new kanji character at the same time as they study a
new word which uses it, so in Japanese study of kanji is very similar and ultimately boundto study of vocabulary. is thesis does not propose any revolutionary new method forChapter 1: Introduction

3

studying kanji and words, but instead aims to help learners better utilise their existing study
methods by modelling the implicit relationships between diﬀerent kanji due to their visual
similarity. is visual similarity is then available as a resource which can aid not only in
dictionary lookup – as in Jane’s case above – but also in evaluating a learner’s proﬁciency.
is thesis targets Japanese for two main reasons. Firstly, Japanese is considered amongst
the most diﬃcult languages to learn for learners coming from an English or European language background, comparable only to Arabic, Chinese and Korean.1 is diﬃculty makes
it an attractive target for improvements to study methods, since even a small improvement
may be signiﬁcant over a long enough time span. Secondly, the aspects of Japanese kanji
characters which present the largest obstacles for learners also present the largest opportunities for the use of rich user modelling.
In order to construct confusion models which approximate the human experience ofkanji similarity and confusability, we draw inspiration from psycholinguistic research on
kanji perception, and construct a number of plausible approximations to the human experience. ese models consider the similarity of kanji as generic images, the extent to which
components are shared, the extent to which broad layout is shared, the similarity in hierarchical tree structure, and the diﬀerence between the stroke sequences of two kanji. In
order to choose between these models, we develop three separate datasets on which they
can be evaluated. ese data sets span a range of authentic sources, from the layman’s raw
judgements of kanji similarity to expert opinions on kanji confusability. By evaluating ourmodels over these data sets, we ﬁnd that the two models that best match human similarity
judgements are tree edit distance and stroke edit distance.
As an initial application for these similarity models, we show that the FOKS (Forgiving Online Kanji Search) dictionary interface provides a useful test-bed for new searchtechniques. After re-engineering the dictionary into a more modular state, we convert oursimilarity models into confusion models, and use these kanji confusion models as the basis
for search by similar kanji. It is this similarity search which allows queries like that of ourearlier example, where 食い was used to ﬁnd 養い. is signiﬁcantly increases the accessibility of the dictionary by allowing learners to search for unknown kanji using their more
1

From National Virtual Translation Center (2007) estimates of time required to learn a language ﬂuently
for English native speakers.

4

Chapter 1: Introduction

frequent and usually simpler visual neighbours. At the same time, it is able to automaticallycompensate for errors where users unintentionally mistake an unknown kanji for a familiar
one they know, allowing learners to still ﬁnd the word they were looking for. rough log
analysis, we ﬁnd that learners’ use of this new form of search exceeds that of earlier lookup
by pronunciation. Analysis of query and result pairs suggests that, rather than confusingrare items for known items as we had assumed, learners instead tend to confuse pairs ofkanji where both lie on the periphery of their knowledge.
As part of our overhaul of the dictionary, we also rebuild its grapheme-phoneme alignment algorithm, the basis of its ability to model kanji reading errors. By moving to a semisupervised algorithm, we increase its scalability with increasing dictionary sizes. is allows
FOKS to upgrade its core dictionaries as new versions of them become available. We also
extend the coverage of the dictionary to include greater coverage of place names, and improve its usability through better word translations and more transparent behaviour.
Most learners do not have the luxury of an immersive second-language environment
in which to acquire words and test word knowledge through active use. is means thataside from dictionary-mediated attempts at reading, they limited to repetitive drills such as
ﬂashcards as their actual vocabulary study method. Whilst ﬂashcards are a useful tool, they
are not designed to measure proﬁciency over large numbers of words, such as an entire courseor syllabus. For these larger proﬁciency estimates, learners typically use class exams, which
are limited in scope to the class in question. For more rigorous testing and accreditation, theannual Japanese Language Proﬁciency Test ( JLPT) is also available. However, the JLPT
takes time, costs money, and can only be taken once a year. ese obstacles are due toa fundamental limitation with manually constructed pen and paper tests: if the test is tomeasure what it was intended to, it cannot be taken by the same learner twice. is makes
testing for proﬁciency expensive, since each new test requires a large investment of resources.
Since it is expensive, tests with any strength are metered out sparingly to learners.
We show that using models of kanji confusion and kanji (mis)pronunciation, new tests
can be generated automatically and uniquely for each learner. Since every test is diﬀerent, there is no limit to the number of tests a user can take. We argue that by removing
the scarcity from proﬁciency testing, learners are able to more eﬀectively self-evaluate theirknowledge, which in turn allows them to make better decisions about their study techniques.

Chapter 1: Introduction

5

Furthermore, by taking the learner’s responses to earlier questions into account, these tests
adapt to each learner individually, potentially creating more diﬃcult tests which better highlight any ﬂaws in a learner’s knowledge. We do this through a new testing platform, Kanji
Tester, which we develop to showcase the potential of automated test generation. Kanji
Tester models itself on the JLPT, focusing on question types which directly test vocabulary or kanji knowledge. e JLPT is entirely multiple choice, so that it can be objectively
marked; Kanji Tester uses its two main error models to generate plausibly incorrect multiple
choice options, so as to increase the test’s diﬃculty. Furthermore, each user’s knowledge ofkanji is explicitly modelled, and adapts to the user’s responses after each test, so that any
errors or areas of confusion the learner displays are more likely to be shown as options inlater questions.
Analysis of several months of Kanji Tester’s usage logs yields a number of interestingﬁndings. e amount of per-user adaption was limited by data sparsity issues, suggestingthat future systems should model errors across groups of users, for example across all usersof the same ﬁrst language background, or else ensure that learners revisit error cases more
frequently. Despite this limitation, we ﬁnd that adaptive test questions based on reading
are signiﬁcantly more diﬃcult than those of our control baseline, due largely to the strengthof the priors used in the adaptive models. Analysis of power user responses shows a weak
positive correlation between ability and period of use, although most power users had highinitial proﬁciency estimates, suggesting that scope for their improvement was limited.
rough this combination of enhanced dictionary support and improved, repeatable
and accurate learner testing, this thesis supports learners of Japanese in their self-study ofvocabulary, and ultimately in reaching their desired level of proﬁciency.

1.1

Aim and contributions

e primary aim of this research is to establish improved software-supported methods
for self-study of vocabulary. In particular, we focus on Japanese as a language whose extreme
orthographic depth means that learning to read well is diﬃcult, and whose logographic
characters oﬀer a rich and under-utilised resource to draw from in aiding learners.
is thesis makes three main contributions. Firstly, it deﬁnes ﬁve separate models of6

Chapter 1: Introduction

graphemic similarity of Japanese kanji, establishes three separate data sets of human judgements, and evaluates the models over these judgements. It ﬁnds that stroke and tree edit
distance metrics outperform other similarity models over these data sets. Secondly, it transforms these similarity models into models of plausible misrecognition of kanji, which in turnare used to provide a novel form of dictionary search by similarity. rough post-hoc log
analysis, it demonstrates that this form of search is both viable and highly useful to learners.
irdly, it combines kanji confusion models with existing models of plausible misreadingin order to generate vocabulary tests for quick self-evaluation of proﬁciency. It ﬁnds that
this modelling allows creation of questions which are signiﬁcantly more diﬃcult than thoseof a control baseline.
A general theme of this thesis is the use of careful error modelling to support learners intheir study. ese error models are used ﬁrstly to compensate for user errors in a dictionary
search application, then to explicitly provoke user errors in a testing application. In bothcases, we argue that learners are better served by the heightened awareness of their behaviour
that these applications display.

1.2

Audience

is thesis is aimed at a general academic audience with an interest in second language
learning, in particular for East-Asian languages. Prior knowledge of the Japanese and Chinese writing systems is helpful but not required. However, several chapters perform formal
modelling or evaluation which require some basic knowledge of probability theory, metricspaces and uses concepts from information retrieval. We nonetheless expect a non-technicalreader to be able to gloss over these parts and still gain a good understanding of this work.

1.3

esis structure

e remainder of this thesis is broken into six chapters as follows:
Chapter 2: e Japanese Writing System
We provide an overview of the Japanese Writing System, discussing its origins, theChapter 1: Introduction

7

three main scripts used, and in particular the kanji script. Issues relating to computer
input and dictionary lookup of Japanese are covered, before concluding with a brief
overview of four separate word formation eﬀects which complicate word pronunciation.
Chapter 3: Learning Japanese vocabularyis chapter ﬁrstly makes the case that vocabulary study is the largest barrier for language learners, and argues that the majority of vocabulary is acquired through autonomous self-study. It examines in detail what it means to know a word, and thus
demonstrates that the word is an appropriate lens through which to view second language acquisition.
We investigate and discuss strategies for deciding what words to study, and then how
to study them. We then consider how words are accessed and stored in the mental
lexicon, and relate these patterns to the general lexical relationships which form between words. We introduce the concept of graphemic neighbourhoods as the relationof near-homography, and situate it amongst other better known relationships such as
near-synonymy and homophony. Finally, we explore the state-of-the-art in dictionary lookup for Japanese, in particular focusing on methods for input and lookup of
kanji characters. We discuss the role of testing in second language learning, and consider the gradual transition from paper-based testing to computer-adaptive testing,
before examining the current forms of tests actually used in the ﬁeld.
Chapter 4: Orthographic similarity
We investigate the notion of graphemic similarity and propose several formal models: a cosine similarity over radical vectors, designed to capture the salience of radicals
as signiﬁcant subcomponents; the L1 norm over kanji images, attempting to measure
visual similarity in a raw and unbiased way; the tree edit distance between kanji structure representations, attempting to compare layout information; and, an edit distanceover stroke vectors, which tries to both capture the salience of stroke order, and to
perform a fuzzy matching of larger components.
ree separate data sets are used to evaluate these metrics: a simple experiment elic-

8

Chapter 1: Introduction
iting similarity judgements from learners, a series of expert judgements mined froma commercial ﬂashcard set and an experiment eliciting native-speaker judgements ofconfusability. Evaluating against these data sets, we ﬁnd that tree edit distance andstroke edit distance models achieve best performance, and do comparably well to one
another. Since stroke edit distance is two orders of magnitude faster than tree edit
distance to calculate, we establish it as the preferred metric given available data.

Chapter 5: Extending the dictionary
We examine the current contribution of the FOKS dictionary interface to the Japanese
lookup space, and assess its suitability as a test bed for new forms of dictionary search,
ultimately rebuilding it to serve this function. Its unsupervised grapheme-phonemealignment algorithm is examined and replaced with a more scalable semi-supervised
method using a dictionary of kanji readings. e resultant improvement in scalability allows our reimplementation of FOKS to utilise the newer and larger dictionaries
versions becoming available over time.
e usability of the system is then enhanced in several ways: its coverage of proper
nouns is extended through the construction of a simple place name gazetteer, dictionary translations are improved to display richer word information, and error correction is made transparent to users through a query explanation tool.
Finally, and most signiﬁcantly, we use the stroke edit distance developed earlier topower a confusability model for Japanese kanji, based on the assumption that learners
will confuse rare kanji for their more frequent neighbours. We use this model to
implement search-by-similarity in FOKS. A theoretical argument is made concerning
the potential improvement in accessibility that this new form of search can provide
for learners. Post-hoc analysis then ﬁnds that over the log period analysed, searchby-similarity was used more frequently by learners than FOKS’s existing intelligent
reading search, demonstrating the utility of this method.
Chapter 6: Testing and drillingis chapter investigates the use of error models to generate randomised vocabulary
tests through a new system called Kanji Tester, which serves both as an improvedChapter 1: Introduction

9means for learners to rapidly self-evaluate and as an alternative and more constrained
platform for evaluating our error models. It proposes that proﬁciency tests can besorted by their scope and their availability, and situates the ideal proﬁciency test, the
JLPT tests, and our current work along these scales. It then discusses in detail theform of the JLPT test, giving examples of the question types that Kanji Tester attempts to emulate.
We then discuss Kanji Tester itself in detail, both in its interface from a user’s perspective, and in its extensible architecture and underlying modelling from our perspective.
Amongst the various forms of potential user models Kanji Tester could use, we discuss and justify our use of per-user per-kanji error models. We then describe ouralgorithm for generating questions, and for updating each user’s error models.
In our evaluation, we discuss diﬀerences between human constructed tests and theautomated tests generated by Kanji Tester. A weak positive correlation is found between test scores and time over which Kanji Tester is used. rough an analysis ofrater responses we ﬁnd that, despite issues with data sparsity, our adaptive test questions cause signiﬁcantly more errors than those of a control baseline.
Chapter 7: Conclusion
Finally, we summarise the thesis and identify three main areas for future work. Firstly,
we suggest potential improvements to the graphemic similarity models this thesis has
proposed, and useful experiments which could be performed to better understand the
topologies they generate. More generally, we identify in this discussion further areas of interest in the modelling of lexical relationships. Secondly, we suggest several
areas in which dictionary accessibility could be improved, including better example
sentence selection, crowdsourcing of dictionary enhancements, and the potential useof semagrams in future open dictionaries. irdly, we discuss the remaining gap between our automatically generated tests and human-constructed tests, and ways ofnarrowing that gap. ese include improved user modelling for the same types ofquestions, and ways to generate new and diﬀerent forms of questions which could
help to test proﬁciency.

Chapter 2
e Japanese writing system
is chapter gives a brief and focused introduction to the Japanese writing system, inorder to provide the background necessary to discuss issues speciﬁc to acquisition of Japanese
vocabulary. Readers already familiar with Japanese may skip ahead to Chapter 3. Likewise,
readers interested in a more comprehensive coverage of the Japanese writing system arereferred to works by Backhouse (1996) and Tsujimura (1999).

2.1

Overview

e Japanese writing system uses three principal scripts, the morpho-syllabic kanji script
and the syllabic hiragana and katakana scripts. e latter two scripts contain the same soundinventory, and so are potentially interchangeable, but in practice each ﬁnds complementaryand largely non-overlapping use. ey encode not full syllables, but a type of short syllable
called a mora. ese scripts are best understood in terms of the division of labour between
them, as suggested by Backhouse (1996:43). Hiragana is principally used for grammaticalelements such as inﬂectional suﬃxes for verbs and adjectives, case markers, grammaticalnouns, amongst others. Whilst katakana encodes the same sounds, its use is likened to thatof italics in English to mark a word for special attention. It ﬁnds principal use in writing
loanwords or names from English or European languages. ese two syllabic scripts areusually learned in full in a short period of time, and are not a signiﬁcant challenge forlearners. We now discuss the more complicated kanji script.
10Chapter 2: e Japanese writing system

11

Kanji are used at the morpheme level in Japanese, and are used to write most lexicalised
words, such as 家 uchi “house, family”. Verbs and conjugating adjectives typically use kanji
to represent the ﬁxed stem of the word, and hiragana for the changing inﬂectional suﬃx
(Backhouse 1996:51), as in 行く iku “to go”. Kanji in context have readings of one ormore mora, but this reading need not cover the whole stem, as shown by examples such as
厳しい kibishī “strict”. Furthermore, the stem itself may not end on a mora boundary: ourearlier example 行く iku has stem ik-, with inﬂected forms such as 行き iki and 行け ike.
As a generalisation, kanji which form standalone words have more concrete meanings,
whereas kanji which form bound morphemes only take on concrete meanings with the addition of either other kanji to form a compound or hiragana inﬂection. e senses of anindividual kanji can be considered its semantic contribution to the many compounds andwords in which it takes part, where such a contribution is transparent. ese senses in turnare often extensions of a single core meaning. For example, Halpern (1999) lists glosses forseveral senses of kanji 暴 bō:
1. violent, rough, wild, cruel, tyrannicalas in 暴力 bōryoku “violence, force”
2. sudden, abrupt
as in 暴落 bōraku “slump, crash (in prices)”
3. unrestrained, inordinate, wild, excessive, irrational
as in 暴飲 bōiN “heavy drinking”
e core meaning given is “violent”.
Kanji characters were historically borrowed from Chinese – whose characters we callhanzi – and most Japanese characters are also valid characters in Traditional Chinese. However, both Chinese and Japanese have undergone some simpliﬁcation, and these simpliﬁed
forms are not necessarily shared. e character etymology for 歳 sai “year, age” in Figure 2.2
is a common example of this process. We call two characters which share the same historical roots cognate. Despite more extensive character simpliﬁcation in Chinese, this cognate
relationship between character pairs means that they typically remain similar in form, andindeed are often identical. However, the same can not be said for their pronunciation.

Chapter 2: e Japanese writing system
12

漁ﬁre /ka/ﬁeld /ta/(wrap)

ﬁshing /asa(ru)/ﬁsh /sakana/

水 魚
water /mizu/勹田火

Figure 2.1: Exposing the compositional nature of kanji. 漁 asa(ru) “ﬁshing” is made by combining 水 mizu “water” (via shortform 氵) and 魚 sakana “ﬁsh”. Further decomposition of 魚 is possible, however its subcomponents do not have a semantic
relationship with its whole; this is because 魚 is a pictograph (i.e. it is derived from a picture of a ﬁsh rather than from thesecomponents).

Chapter 2: e Japanese writing system

13

Each Japanese character, kanji or otherwise, occupies an identically sized square spacewhen printed or hand-written. For example, a simple kanji such as 工 is given the same
space as a more complex kanji like 蠹. is can be contrasted with other writing systems,
for example the latin alphabet, where each letter may occupy a diﬀerent amount of space.
Kanji are also compositional, so that many basic kanji also occur as components of more
complex kanji. An example of this composition is given in Figure 2.1, where we decompose
漁 asa(ru) “ﬁshing” into its constituent parts 水 mizu “water” and 魚 sakana “ﬁsh”. Notethat to preserve space, 水 is given in its short form 氵; a number of kanji have a short form
radicals for composition in this manner. In this case, both “water” and “ﬁsh” are semanticallyrelated to “ﬁshing”, so 漁 is a good example of reasonably transparent semantic composition.
Note that 魚 itself is composed of radicals 勹, 田, and 火 (via short form 灬), with meanings“wrap”, “ﬁeld” and “ﬁre” respectively, but these meanings seem to bear no relation to theoverall meaning “ﬁsh”. is semantically opaque example occurs since 魚 is a pictograph,
historically derived from a picture of a ﬁsh rather than from these component radicals.
Each kanji may have multiple pronunciations which it takes in diﬀerent contexts, andto explain this it helps to brieﬂy discuss how these readings occurred. Kanji were originally borrowed from Chinese during several diﬀerent periods in history when diﬀerent
Chinese dialects were prominent, so each individual kanji may have diﬀerent pronunciations borrowed from any or all of these periods. ese borrowed readings are known as on
readings (from 音 oN “sound”), and are the source of the limited similarity in pronunciation between cognate characters and words in Japanese and modern day Chinese. An on
reading may be indicated visually by a phonetic component within a kanji, such as 同 dō, asin 銅 dō “copper”, but may also occur without visual cues. In contrast, kun readings (from訓 kuN “explanation”) are native Japanese pronunciations which are never visually indicated,and must simply be learned.1
As a broad generalisation, on readings are used for kanji when combined with other kanji
into compounds, whereas kun readings are used in combination with hiragana inﬂectional
suﬃxes. Native speakers learn both for common characters, and can distinguish between
1

Note that for transliteration of Japanese linguistic examples we use the Hepburn system with Backhouse style “N” mora. is allows us to distinguish between the transliterations of 勲位 kuNi “order of merit”(segmented as ku-N-i) and the 国 kuni “country” (segmented as ku-ni). However, when borrowing linguistic
terminology like kun from Japanese, we use the standard Hepburn system for simplicity.

14

Chapter 2: e Japanese writing system

reading types. e frequency distributions of each reading type also diﬀer, naturally reﬂecting the languages and dialects they originated from. In extreme examples, koN only occurs
in Japanese as an on reading, but 154 kanji may take this pronunciation, including 金, 今and 近; aki is only used as a kun reading, but 108 kanji may take it, including 日, 明 and 成.
If we compare with Chinese, we ﬁnd that Chinese only uses the hanzi script, and its major dialects have diﬀerent sound inventories to Japanese. Furthermore, each hanzi typicallyhas only one valid pronunciation in each dialect, which may in turn be cued by phonetic
radicals where they occur. is substantially simpler form-to-sound mapping means thatwork based on Japanese pronunciation is not likely to be directly useful for Chinese. However, where this thesis concerns itself with the visual form of single kanji and of multi-kanji
compounds, we expect such work to generalise gracefully.
Our examples of koN and aki also indicate the extreme level of homophony which isfound in Japanese at the character level, though this is reduced at the word level. In spoken
language, context and reading frequency provide suﬃcient tools to disambiguate a kanji orword. In written language, the use of kanji makes this eﬀect negligible: a kanji’s pronunciation is reﬂected in its visual form in a limited manner, so homophony and homography need
not overlap. A clever tongue-twister takes advantage of this. にわにはにわにわとりがいる is pronounced niwa niwa niwa niwatori ga iru, and is extremely ambiguous if either spoken aloud or written in hiragana due to its use of several homophones. However, when kanji
are used it is written as 庭には二羽鶏がいる and contains no semantic ambiguity, meaning “there are two chickens in the garden”. e comparable Buﬀalo buﬀalo Buﬀalo buﬀalo
buﬀalo buﬀalo Buﬀalo buﬀalo sentence in English, comprised exclusively of three diﬀerent
senses of the word buﬀalo, retains its ambiguity in written form (Pinker 1994:210). isconcisely illustrates the utility of kanji in disambiguating homophones during reading.

2.2

Input and lookup of JapaneseComputer input in Chinese and Japanese is markedly diﬀerent from that of languages
such as English with alphabetic scripts, since there are far too many diﬀerent charactersto admit any simple mapping from the available keys on a keyboard to characters a user
may wish to write. For this reason, a sequence of keystrokes is required to build an inputChapter 2: e Japanese writing system

15

Oracle characters

Bronze characters

Seal script!

Traditional Chinese"

Simplified Chinese!

Japanese

Figure 2.2: e etymology for the character 歳 sai “year, age” from oracle characters tomodern day usage in Chinese and Japanese, and the cognate relationship between the two
usages. Cognates are from Goh et al. (2005), and early character forms are from Sears(2008).

16

Chapter 2: e Japanese writing system

space large enough to map to any character. is led to the use of input method editors(IMEs), programs which translate a sequence of keystrokes into a shorter sequence of more
complicated characters, usually interactively. Languages with alphabetic scripts also use an
IME when the keyboard size is restricted, for example the “predictive text” feature on mostmodern mobile phones.

IME use for typing in Japanese

1. Type in pronunciation, hit return

2. Select correct form from options

Figure 2.3: An example of IME use to type in Japanese. e pronunciation is ﬁrst
entered, then matching word forms are displayed to choose from. e example reads
カレーを食べよう karē o tabeyō “let’s eat curry”
In Chinese, where only one script is used,2 there is a considerable variety of methods
used by IMEs to constrain the space of characters down to the character the user was looking for. Ultimately, this is achieved based on cues about the character’s form, as in the
Wubizixing (五笔字型 [wǔbǐ zìxíng]) or Cangjie (仓颉 [cāngjié ]) methods, or based on
the character’s pronunciation, as in the Pinyin (拼音 [pīnyīn]) or Zhuyin (注音 [zhùyīn])
methods. For both Chinese and Japanese languages, the large amount of homophony meansthat input by pronunciation is best disambiguated in word-level chunks, since the additional
context helps to constrain possible matches.
2

Actually, modern Chinese characters occur in both simpliﬁed and traditional forms, as shown in Figure 2.2. Since each dialect predominantly uses one script or the other, we can call Chinese languages singlescript languages.

Chapter 2: e Japanese writing system

17

Both forms of input methods suﬀer the same basic problem: when based on form, theuser is unable to type a character whose form they have forgotten or never learned; whenbased on pronunciation, the user is unable to type a character whose pronunciation they
have forgotten or never learned. Since Japanese contains the two syllabic kana scripts, its
input methods are based on pronunciation, and Japanese input thus faces the latter problem. However, the problem is worse for Japanese, since the user cannot easily change input
method when faced with a problematic character, as graphemic input methods are not readily available for Japanese.3
Compounded with this, kanji have much weaker cues to pronunciation than hanzi. Recall that some kanji have reliably phonetic radicals which provide a known on reading, as in
our earlier example 同 dō, as in 銅 dō “copper” and 胴 dō “body”. However, even in thesecases, only one of many potential readings is known. For example, 同 can also be read ona(as in 同じ onaji “same”) and 銅 can also be read as okagane. In most cases there is no reliable phonetic to suggest the pronunciation of an unknown kanji, and thus no easy meansof computer input.
An appropriate response to this fundamental problem for Japanese IMEs is a change
in input modality. is is attempted by the many handwriting recognition interfaces forJapanese: dictionaries for the Nintendo DS,4 Todd Ruddick’s web-based recognition applet,5 Tomoe6 and many commercial electronic dictionaries. However, of the computerbased systems, many still suﬀer from several issues: the awkwardness of mouse input for
drawing characters; sensitivity to both stroke order and connectivity of components; and thediﬀerence in hand-writing styles between learners and native speakers. On the other hand,such technology has improved markedly in the past few years. For example, Ben Bullock’s
new hand-written interface7 is signiﬁcantly more robust than earlier interfaces in terms ofthese issues. Overall, whilst such systems are not always optimal, they provide a very useful3

Phonemic input methods are reasonably natural, since input is a simple representation of speech. However, graphemic input methods require some more complicated mapping scheme, which takes time and training to use. Furthermore, the two additional syllabic scripts in Japanese are far better suited to phonemic input
methods.
4
For example, 漢字そのまま楽引辞典 kaNji sono mama rakubiki jiteN.
5
http://www.csse.monash.edu.au/~jwb/hwr/6
http://tomoe.sourceforge.jp/
7
http://kanji.sljfaq.org/draw.html

18

Chapter 2: e Japanese writing system

alternative in situations where other input methods are too slow.
e Kanjiru dictionary takes a diﬀerent approach, adapting Wills and MacKay’s (2006)
Dasher accessibility interface, attempting to interactively assemble a character by shape andstroke in an adaptive manner (Winstead 2006; Winstead and Neely 2007). e user guides
the search through mouse movements, providing the user with a way of building up components stroke by stroke until the desired character is found. ough cumbersome for typicalinput, it provides an alternative version of input-by-form for diﬃcult characters.
Despite this arsenal of lookup and input techniques, inputting or looking up an unknown kanji or word remains a barrier for learners, as indicated by the continuing emergence of new lookup methods. We revisit the topic of dictionary lookup in more detail in
Section 3.3.

2.3

Word formation in JapaneseJapanese is an agglutinative language, which can present additional diﬃculties in appropriately determining word-hood. New words in Japanese are principally created throughone of several mechanisms: borrowing from foreign languages, compounding of existingwords, modiﬁcation of existing words, or clipping of existing words to create more colloquial forms (Backhouse 1996:81). Since most loanwords are written in katakana, which isphonetically transparent to learners, we largely ignore word formation from borrowing inthis thesis. In later chapters we will show that many of the errors which learners make in
dictionary lookup and testing relate to a lack of transparency in the compounding process
by which many words are created. We thus also ignore word formation by modiﬁcation orclipping in order to focus better on aspects of the compounding process.
In later parts of this thesis it will become important to be able to reverse the compounding process, decomposing a compound back into its smallest constituent morphemes. In
Japanese, this task is essentially one of Grapheme-Phoneme alignment, which we consider
in earnest in Section 5.2. Here, we discuss brieﬂy four pertinent features of word formationby compounding and its reversal: okurigana, sequential voicing, sound euphony and grapheme
gapping.

Chapter 2: e Japanese writing system

19

OkuriganaSingle kanji do not always correspond to whole morphemes in written language, often
requiring a hiragana suﬃx to form a unit. is is especially true for kanji forming verbs ori-adjectives, where the hiragana suﬃx forms an inﬂectional ending (Backhouse 1996:43).
For example, 行 is not a word by itself, but with suﬃx く ku forms the verb 行く iku “to go”.
In general, these suﬃxes are called okurigana, and useful alignments should include themalong with their kanji stem in order to preserve the basic morpho-phonemic structure of
the compound.
Although most cases of okurigana represent verb and adjective conjugation, there aremany general cases such as that of the kanji 取, which occurs in compounds almost exclusively with suﬃx り ri as 取り tori, but is sometimes written with the suﬃx ri conﬂatedwith the kanji stem. For some words, the written form can occur either way. For example, 受け取り uketori “receipt” can equally be written 受取り, 受け取 or 受取, dependingon whether the two hiragana け ke or り ri are conﬂated into their respective kanji stems.
Ideally, alignment systems should capture such alternations in order to achieve consistentsegmentation behaviour. is is also useful for attaining an accurate estimate of the frequency with which a given kanji occurs with a particular reading which is independent of
the exact lexical form of the word.
Sequential voicing
Sequential voicing occurs when the second component in a compound has its initialconsonant changed from an unvoiced sound to a voiced sound (Backhouse 1996:82). For
example: 本 hoN “book” + 棚 tana “shelf ” → 本棚 hoNdana “bookshelf ”. Its occurrence islimited by a number of constraints. For example, it is mainly restricted to native words, andoccurrence is further constrained by Lyman’s law, which states that sequential voicing will
not occur where there are existing voiced obstruents in the tailing segment (Vance 1987).
It occurs in about 75% of cases where Lyman’s law is not violated, with some systematic
irregularities for noun-noun compounds (Rosen 2003).
Alignment methods based on precedence or frequency counts may be hindered by sequential voicing, since aligned grapheme/phoneme pairs may not be recognised as phono-

20

Chapter 2: e Japanese writing system

logical variants of previously seen kanji–reading pairs. Fortunately, devoicing is a relatively
simple 1-to-1 process, so a common approach is to simply consider voiced and devoiced
grapheme/phoneme pairs to be equivalent for counting or comparison.
Sound euphonySound euphony occurs when the ﬁrst component’s last syllable alters to match the soundof the tailing segment. is creates a syllable-ﬁnal geminate consonant which is pronounced
for audibly longer than a normal short consonant (Backhouse 1996:26). In Japanese, gemination is indicated by the small っ character, and is typically romanised into double consonants, such as in まって ma e “wait” and いっぱい ippai “full”. An example of soundeuphonic change is in the combination of 国 koku “country” and 境 kyō “boundary” to create
国境 kokkyō “national border”.
Unlike sequential voicing, which imposes an easily reversible transformation, it is notclear from 国境 kokkyō “national border” what the original pronunciation for 国 was (possibilities include koki, koku, kosu and kotsu). is can introduce some diﬃculty in reversing
the compounding process and determining correctly the pronunciation of the original parts.
Grapheme gapping
We saw earlier that in cases of okurigana, part of the reading for some words can be optionally conﬂated into the kanji. Grapheme gapping refers to a much rarer occurrence wherea particle which is a standalone morpheme is subsumed into a kanji compound. Typically insuch cases it is also acceptable to write the particle explicitly. For example: 山 yama “mountain” + の no “GENITIVE” + 手 te “hand” can be written as either 山手 or 山の手, bothwith identical pronunciation yamanote. In the ﬁrst case the particle no is subsumed as partof the compound; in the second it is explicitly written. Grapheme gapping is very rare,normally only occurs with the particles ga or no, and tends not to be productive, suggestingthat even high-precision alignment systems need only store individual known cases.

—

Chapter 2: e Japanese writing system

21

With this focused overview of the Japanese writing system complete, and in particularhaving discussed the basic problems for learners associated with the kanji script, we can turnto the main theme of this thesis, vocabulary acquisition.

Chapter 3
Learning Japanese vocabulary
In this thesis, we focus on improving study tools for learners of Japanese so that they may
better acquire vocabulary and ultimately, improve their Japanese proﬁciency. is chapter
provides the motivation for our approach, and then extended background into each of the
focuses for later chapters.
In Section 3.1 we ﬁrstly situate this thesis within the debate on how to best facilitate
second language acquisition. We argue that supporting learners in their self-study of vocabulary is likely to yield the greatest beneﬁt. We then examine current self-study strategies
and tools, focusing on Japanese as a target language, and argue that better modelling of user
errors could greatly improve their eﬀectiveness, especially for learners at the crucial earlyreading phase.
In order to guide this error modelling, Section 3.2 examines current research as to thestructure of the mental lexicon to try to determine how words are accessed during reading.
We show that the access structure for Japanese kanji is suggestive for how we might model
misrecognition errors and error models based on visual similarity.
Since dictionary lookup and language testing are arguably the two most fundamental
mechanisms for second language learning, we go on to examine each in turn. Section 3.3
ﬁrstly discusses the current state of dictionary lookup for Japanese from the perspective ofboth readers and writers, and then Section 3.4 considers recent advances in language testing.
ese sections provide the necessary background for Chapters 5 and 6 respectively.
In discussing vocabulary, this chapter uses the basic unit of vocabulary, the word, as a22

Chapter 3: Learning Japanese vocabulary

23

lens through which to consider language acquisition. However, diﬀerent ﬁelds of researchuse diﬀerent deﬁnitions of “word”. In this thesis, we use “word” to mean lemma, a headword
in canonical form from which inﬂected forms can transparently be derived. For example, we
consider change, changes, changed and changing to be forms of the same word change, rather
than separate words in their own right. Where we refer to research which uses an alternative
deﬁnition, such as the narrower meaning of token or the broader meaning of word family,
we use the more speciﬁc terminology instead.
Whilst we recognise the strong role of memory in word knowledge, a proper treatmentof this broad issue is beyond our scope. is chapter examines in detail how words areaccessed and many aspects of how they are best stored, but largely ignores longer-term issuesof ensuring that words learned stay learned. Readers looking for a more comprehensive
discussion of the role of memory in language are referred instead to works by Baddeley(1997, 2003).
A ﬁnal caveat to our discussion is that, despite referring to relevant Chinese- or Japaneselanguage work that we are aware of, our coverage of academic publications in these languages
is regrettably more limited than that of English-language publications. is has little eﬀect
on our main focus of second language learning, since such work is multilingual by nature,
however it does restrict our view of native speakers of these languages. Detailed aspects ofnative speaker development particular to Chinese and Japanese are thus beyond the scopeof this thesis.

3.1

Autonomous vocabulary acquisition

e beginner’s paradox
Vocabulary is signiﬁcant, above all other areas of language proﬁciency, because it isopen-ended. New product names, person names, and word forms are created every day,and a native speaker will continue to acquire words throughout their entire life. is openended nature makes learning suﬃcient vocabulary a heavy burden on second language learners. Frequency and redundancy allow communication with a limited lexicon, but for ﬂuent
communication learners must still develop a lexicon of suﬃcient size.

24

Chapter 3: Learning Japanese vocabulary
So how do learners acquire such a lexicon? For ﬁrst-language (L1) learners, our answer

has two parts. By the age of ﬁve, L1 English children know some 3000-5000 “word families”, covering most of spoken language, without direct instruction (Nation 2001:96). esource of this initial oral vocabulary is debated, but they are presumed to have learned it
through exposure to adult speech. Yet between the ages of 8 and 18, they similarly continue
to learn between 7 and 15 words per day, depending on how we count (Landauer and Dumais 1997; Vermeer 2001). Furthermore, since about three quarters of an English speaking
L1 adult vocabulary occurs almost exclusively in written text, it must have been acquired
through reading, as suggested by Miller (1941). e vocabulary acquired this way seems
to be the “long tail” of language, made up of large numbers of medium-to-low frequency
words. Current evidence thus suggests that in an L1, vocabulary is acquired autonomously,
and through reading widely.
In a second language (L2), Krashen (1989) similarly argued that reading widely is thesource of most vocabulary acquired. However, in an L2 this is premised on having alreadyacquired the early core vocabulary. is is reﬂected in high-level reading skills which only
begin transferring from a learner’s L1 to their L2 at around the 3000 word family mark
(Laufer 1997:24), and also in Hsueh-Chao and Nation’s (2000) study of unknown word
density which suggested 98% of word tokens in a text should be known for reading to be
comfortable. Although graded readers1 can be used to achieve this at lower vocabularylevels, learners must still acquire a signiﬁcant amount of vocabulary before they can expect
success in a reading program. Indeed, many factors may play a part in the eﬀectiveness ofreading widely for L2 vocabulary learning, and several studies such as Coady (1997:226)
and Fraser (1999) describe mixed results. Landauer and Dumais (1997) provide a potential
explanation for some of these results, suggesting that word knowledge is built graduallyfrom large numbers of weak links with other words. If this was the case, then much of the
vocabulary improvement from reading a series of text would appear in words not explicitly
encountered in the texts, but rather whose meaning has solidiﬁed through relationships
with encountered words. is suggests a potential source of measurement error for studies
1

A graded reader is a text where words assumed to be too diﬃcult for the reader are either pre-glossed or
replaced by simpler near-synonyms. In this way a text is constructed with an appropriate density of unknown
words for the reader.

Chapter 3: Learning Japanese vocabulary

25

of reading that look for vocabulary growth only in explicitly encountered words.
Nonetheless, some controversy remains amongst experts as to the full role of reading in
L2 vocabulary growth. In this thesis, we make the reasonable assumption that L2 learners
achieve the same vocabulary growth from reading as L1 learners given similar vocabularysize, reading materials, and opportunities for input. Where diﬀerences exist, we assume theylie in background, environmental or strategic factors rather than the L1/L2 distinction.
Vocabulary knowledge promotes reading comprehension: it is the greatest predictorof reading success, better than general reading ability or syntactic ability (Laufer 1997;
Urquhart and Weir 1998). e converse is also clear, that reading comprehension promotes vocabulary growth, since strong context allows learners to better infer the meaningof unknown words. ese two mutually enforcing factors seem to form a positive feedback
loop, which could form a strong platform for reaching ﬂuency.
Nuttall (1996:127) however found that there are not one but two feedback loops, depending on a learner’s attitudes, motivation and existing reading proﬁciency: a “virtuous
circle” reinforcing success and a “vicious circle” reinforcing failure (Figure 3.1). ese circles are so-named because starting at any of the points leads to each of the other points inthe cycle. Pulido and Hambrick (2008) in turn validate these ideas through empirical studyof L2 reading. Coady (1997) calls this problem the “Beginner’s Paradox”, since learners
must acquire vocabulary through reading, but need suﬃcient vocabulary to bootstrap theprocess.
If we are serious about helping learners to reach higher levels of ﬂuency, we must focus
our eﬀorts where the long-term impact will be the greatest. is is not a new idea: where
the appropriate strategies are not developed automatically, many researchers now advocate
teaching autonomous learning strategies explicitly, at the cost of time spent explicitly teaching language, because of the beneﬁcial eﬀects of these strategies on long-term autonomous
behaviour (Huckin and Coady 1999; Barker 2007). Use of appropriate tools is also increasing; Tozcu and Coady (2004) found that teaching high-frequency words using a CALL
system improved reading more than simple reading practice. is thesis describes attempts
to improve tool support for learning vocabulary at the early reading stage, in order to ensureleaners achieve a positive cycle of vocabulary growth in their reading.

26

Chapter 3: Learning Japanese vocabulary
Vicious circle

Virtuous circle

Doesn'tunderstandEnjoysreading

Doesn't read
much

Reads
slowlyDoesn't
enjoy
reading

Understands
betterReads faster

Reads more

Figure 3.1: Nuttall’s (1996) cycles of frustration and growth in L2 reading.

Knowing a word
In order for a learner’s vocabulary size to be accurately measured, we need a sense ofwhat it means to acquire a word. We settled earlier on using “word” to mean lemma. To
determine what we might mean by acquire, we ask, what does it mean to know a word?
Nation (2001:27) attempts to provide an exhaustive listing of dimensions of word knowledge. Each aspect ﬁts under the broad categories of form, meaning and use, and furthermore
distinguishes between the knowledge required to recognise a word (receptive knowledge[R]) and that required to produce the word correctly (productive knowledge [P]). Using this
distinction, Table 3.1 attempts to exhaustively describe the dimensions of word knowledge.
is enumeration is important since it provides an implicit deﬁnition of depth of vocabularyknowledge: the depth of knowledge of a word is the learner’s coverage over these aspects ofknowledge. It also serves as a reminder that word knowledge does not exist independently
in a vacuum, but in fact cross-cuts almost all aspects of linguistic proﬁciency.
Most studies into vocabulary acquisition have used languages with alphabetic orthographies (in particular English, French and German) as their L2. So what diﬀerences could we
expect with languages like Chinese and Japanese which use more complex scripts? At theChapter 3: Learning Japanese vocabularyForm

spokenwrittenword parts

Meaning

form and meaning
concept and referents
associations

Usegrammatical functions
collocations

constraints on use(register, frequency)

R
P
R
P
R
P
R
P
R
P
R
P
R
P
R
P
R
P

27

What does the word sound like?
How is the word pronounced?
What does the word look like?
How is the word written and spelled?
What parts are recognisable in this word?
What word parts are needed to express the meaning?
What meaning does this word form signal?
What word form can be used to express this meaning?
What is included in the concept?
What items can the concept refer to?
What other words does this make us think of?
What other words could we use instead of this one?
In what patterns does the word occur?
In what patterns must we use this word?
What words or types of words occur with this one?
What words or types of words must we use with thisone?
Where, when and how often would we expect to meetthis word?
Where, when and how often can we use this word?

Table 3.1: Aspects of word knowledge, marked “R” for receptive knowledge and “P” forproductive knowledge. From Nation (2001:27).

28

Chapter 3: Learning Japanese vocabularyForm

spokenwritten

P
R
R

Use

constraints on use

RPP

How is the kanji pronounced in this context?
What semantic components are used in this kanji? How dothey contribute to the whole-kanji meaning?
What phonetic components are used in this kanji? Do they
provide any on readings?
What other variants exist for this kanji?
What is the correct stroke order for this kanji?
In what context is it appropriate to use this kanji? Whenshould its kana equivalent be written instead?

Table 3.2: Our proposed aspects of kanji knowledge, in complement to aspects of word
knowledge shown in Table 3.1.

kanji level, many word-level aspects also apply, but some elements such as pronunciations
in context, variants and stroke order are also important. We propose the additional aspectsof knowledge in Table 3.2 as applying to kanji.
Returning to our question of what it means to acquire a word, we are left with two
extremes. A word we have never encountered is clearly not acquired, yet must a learnerknow every aspect of a word (or kanji) to have acquired it? Acquisition is clearly a gradedconcept. In particular, the extent to which a learner knows about a word is considered thedepth of their knowledge, whereas the number of words of which they have some knowledge
is the breadth.
Most vocabulary tests clearly measure breadth of knowledge, or how many words are
known to some basic degree. Is one more important than the other in language learning?
Fortunately, the evidence so far suggests breadth and depth are strongly related. Vermeer’s(2001) study of 4 and 7 year old Dutch children showed high correlations between breadthand depth measures for both L1 and L2 children. We already considered words as nodes
in a highly interconnected lexical network. Vermeer (2001:218) argues that “the denser the
network around a word, the richer the set of connections around that word, the greater thenumber of words known, and the deeper the knowledge of that word.” e more words alearner knows, the better each individual word’s meaning is delineated from other words it isconnected to. Similarly a word’s exact aspects of meaning can only be known in the context
of other words which are related to it. us, increased breadth in vocabulary knowledge
coincides with increased depth, and vice versa, allowing us to side-step the depth issue inChapter 3: Learning Japanese vocabulary

29

applications and use common breadth measures of word knowledge instead.
We now consider the order in which words should be studied to maximise the impactof the words acquired.

Ordering strategies
Given authentic input, most learners quickly run into far more unknown words thanthey have time to study. Since time and attention are ﬁnite resources, they should put eﬀort
into ensuring the words that they do study are those which will gain them the most beneﬁt.
e cost of learning a word in terms of these and potentially other resources is called its
learning burden (Nation 2006; Barker 2007). Once acquired, knowing a word also accrues
some beneﬁt to the learner: the ability to understand its use in context. is section presents
a brief discussion of two main strategies for choosing what words to study next. e ﬁrst isa focus on compositional aspects of words, the second a focus on frequency of occurrence.
We call both ordering strategies.
A focus on composition advocates studying minimal words or morphemes before studying the larger words or multiword expressions in which they occur. is is typical of hownative Japanese children study kanji, learning basic kanji ﬁrst, and then their use as components in more complicated characters. It is also supported by the depth-of-processing
hypothesis for memory, which suggests that the depth (extent of semantic engagement)with which a word is processed determines the success of subsequent recall (Craik and Tulving 1975). Since new words will be mentally linked to their component parts, we expect
greater semantic engagement during their study; they should thus be easier to remember
than words without such links. is eﬀect was also conﬁrmed by Yamashita and Maru’s(2000) study of kanji compositionality and ease-of-learning. We also note that basic kanji
often relate to simple concepts, and that studies in English likewise suggest that such concepts are easier to learn (Rosch et al. 1976). In general, this strategy advocates a so-called“greedy” approach to studying words, in which learners study the easiest of the unknown
words ﬁrst and thus amass a large vocabulary quickly.
e most common criticism of ordering by composition as a strict strategy is that it
advocates delaying the study of common words in favour of much rarer words with known

30

Chapter 3: Learning Japanese vocabularycomponents. Learners may end up with a larger vocabulary, but it may not serve them aswell as a smaller vocabulary of more appropriately chosen words. In contrast, our second
ordering strategy is to study words or characters in frequency order. In a simple form, ﬁxed
vocabulary lists are constructed for learners based the frequency of occurrence of words (orkanji) in some representative corpora of texts. is approach tries to get maximum coverageover new and unseen input by asking learners to study the words most likely to reoccur,
regardless of their diﬃculty. It also assumes both that learners are uniform in their exposure
to input, and that words are uniform in the learning burden they incur.
In practice, there may be less competition between these strategies than supposed. Given
that each new word presents both a potential burden and beneﬁt, Barker (2007) suggests
a more sophisticated combination of these strategies, where learners pay attention to how
often they encounter a word which is novel or on the cusp of their knowledge. ey canthen decide which words to study, trading oﬀ a word’s learning burden against its beneﬁt,
and thus using their time most eﬃciently.
Regardless of the ordering strategy chosen, the words themselves must still be learned.
We now ask how individual words might best be learned.

Acquisition strategies
In Section 3.1, we argued that vocabulary acquisition was largely an autonomous matter.
To brieﬂy recap, although the most obvious strategy is to explicitly teach vocabulary in theclassroom, and doing so has strong advantages for the words which are taught, studentteacher time is ultimately limited. e sheer amount of vocabulary learners must acquireprecludes this as a primary method of study. A consensus has formed that for high frequency
highly polysemous words, explicit teaching is useful, but that the vast majority of words
will be nonetheless be acquired through autonomous study. We now consider two main
strategies for such study: mnemonics and inference from context.
As we discuss these strategies, we recognise that a large number of variables may aﬀect
their success. It is however beyond the scope of this thesis to consider them in full. We
instead refer the interested reader to Mohseni-Far’s (2008) excellent overview of acquisition
variables and strategies, which we draw on in our discussion.

Chapter 3: Learning Japanese vocabulary31

Mnemonics
From an abstract perspective, vocabulary learning is purely a matter of memorising information, so general memorisation methods such as mnemonics are applicable. Mnemonics in language learning are mediation strategies for mentally linking the form of an L2
word with its deﬁnition, or with an approximate L1 match (Mohseni-Far 2008). For example, suppose an English speaker wishes to learn the Japanese word 五 go “ﬁve”. Since
its pronunciation bears similarity to the English word go, they may use this acoustic link
to associate movement with the number ﬁve, perhaps creating a mental image of ﬁve men
going somewhere. When they wish to recall the Japanese word for ﬁve, they remember themental image they have constructed, and through this link recall its pronunciation as go.
is technique is called the keyword method. Note that the association with movement andthe additional details of the mental image are not part of the meaning of the Japanese word,
but simply additional details used to aid its recall.
A popular method for kanji study making use of mnemonics is presented in the “Remembering the kanji” series of books by Heisig (1985), and is known informally as the“Heisig method”. ree main points deﬁne this method. Firstly, Heisig advocates studying a large number of kanji for their meaning only, and ignoring pronunciation. is is aform of ordering strategy, deferring study of speech, with the goal of greatly simplifying
kanji acquisition. Secondly, the kanji studied should be studied in compositional order, astrategy we discussed earlier. Lastly, each individual kanji is studied through a mnemonic
method, where the meanings of kanji-components are tied with the whole-kanji meaning
into a story the learner creates. For example: the kanji 砕 sai “smash” could be remembered by the story: “To smash something you should hit it with a stone (石) nine (九) orten (十) times.”2 is story provides a form of semantic mediation which may aid recall
and increase processing depth. ere have been no formal studies of the eﬀectiveness of
Heisig’s approach, but criticisms focus on the trade-oﬀ suggested by his ordering strategy;
in contrast his mnemonic method is considered to be a useful tool for learners (So 2008).
Mnemonic methods are supported by the depth of processing hypothesis, as referred to in
our discussion of ordering strategies. By creating additional phonetic and semantic links,
2

A user example submitted to the site “Reviewing the Kanji”, at http://www.kanji.koohii.com/.

32

Chapter 3: Learning Japanese vocabulary

the word is being processed at an increased depth, and retention is subsequently increased.
ough generally successful, mnemonic methods have several limitations. Firstly, manyrequire suﬃcient phonetic or orthographic similarity between L1 and L2 words in order thatmediating links can easily be constructed. Secondly, they can be diﬃcult to use for abstract
words. Finally, they only establish links for one aspect of word knowledge, namely formto-meaning links in Heisig’s case, and sound-to-meaning links in our earlier example 五 go.
Beyond the chosen dimension, they have little to oﬀer in the acquisition or memorisationof the many other aspects of word knowledge discussed earlier.
Inferring from context (IFC)
Mnemonic methods are a way of adding depth to the process of memorisation, and linking new words with existing knowledge so as to provide alternative mental access methods
for the new information. An alternative to constructing these links “artiﬁcially” is to attempt
to study words in their authentic context. is way, the text itself provides the related concepts to link to. e “Kanji in Context” series of kanji study books takes just this approach,attempting to provide authentic and rich contexts for each kanji (Nishiguchi 1994). ismethod is most applicable in open reading, when unknown words are encountered naturally in context. In such a case, learners can increase engagement by actively trying to guessthe item’s meaning instead of or as well as looking it up. is technique is called inferringfrom context (IFC), or the “guessing strategy”. e underlying principle of IFC, that themeaning of a word is embedded in the many contexts in which it occurs, is well known as
the distributional hypothesis in Computational Linguistics.3
An example given by Walters (2006) is illustrative of the idea: Typhoon Vera killed or
injured 218 people and crippled the seaport city of Keelung. Suppose that the learner knows
every word in the sentence but crippled. en the context constrains the meaning of theunknown word, allowing the learner to ask, what would a typhoon do to a city? Noticealso that the meaning is not uniquely constrained by this encounter; many closely related
words could work in place of crippled, such as destroyed, devastated or ravaged, each withslightly diﬀerent meanings or usage in other contexts. In other cases, the context may barely3e distributional hypothesis, originally proposed by Harris (1954), is that words which occur in similar
contexts tend to have similar meanings.

Chapter 3: Learning Japanese vocabulary

33

constrain the meaning at all, and guessing may be unhelpful. Nonetheless, over the courseof many encounters in suﬃciently varied contexts, the learner should be able to graduallybuild a more complete understanding of word meaning. Part of this understanding may
come from learning related words and thereby better delineating each word’s meaning.
In its broadest sense, IFC can refer to any situation in which the learner attempts todetermine the meaning of an unknown word based on available cues. ese could include
diagrams, images, topic information, the other words in a sentence, or any other contextual
knowledge available.4 Although IFC as a strategy could refer to many uses and situations,
it is usually described as a strategy to aid reading.
e vocabulary size that learners bring to bear in their reading attempts strongly deﬁnes
their reading experience. As a rough approximation, the relationship between the numberof unique words and text size can be modelled as log-normal for alphabetic orthographies.5
is distribution suggests how often learners are likely to encounter unknown words as their
vocabulary increases in size. ere will thus be stages in aptitude as a learner progresses frommost words in general text being unknown, to one unknown word sense per phrase, to one
unknown sense per sentence, then per paragraph, and so on. Even at a level of several
thousand lexemes, a learner still falls far short of the vocabulary of a native speaker, andthus encounters unknown words far more often.
IFC is somewhat controversial for second language learners for several reasons. Firstly,inferring accurately requires a large amount of local and global context, which may not be
available for every word. Only in rare cases will the word’s meaning be given fully and redundantly in the course of the sentence and surrounds; this is usually called pregnant context. In
normal cases, there may not be enough information to fully constrain the word’s meaning,
especially if the context itself contains other unknown words. us IFC is most accurate
for native speakers, and becomes less and less accurate with decreasing vocabulary sizes. In
reality, this common argument is not restricted to IFC and L2 learners, but based rather
on the amount of existing vocabulary knowledge. As we discussed earlier in Section 3.1,
4

Normally this strategy refers to inference from context within receptive materials, for example written
or audio-visual materials. In dialogues, where a learner is better placed to negotiate meaning and attempt
clariﬁcation, it is less necessary.
5
See Baayen (1993) for a comparison of log-normal, inverse Gauss-Poisson and generalised Zipf ’s “laws”
on modelling this distribution.

34

Chapter 3: Learning Japanese vocabularyattempting to read widely with insuﬃcient knowledge may be a diﬃcult and unproductive
exercise, causing the learner to enter a cycle of poor reading ability and poor motivation
which is diﬃcult to break from.
Secondly, Dycus (1997) argues that unknown words are typically rarer in usage than
known words, hence carry more information (by information-theoretic measures) than the
known words, and are thus harder to guess. Unknown words are certainly likely to berarer than known words: if we assume that learners build knowledge about new words
through encounters with those words, a corollary is that unknown words are in general
those encountered the least. For this reason, their exact meaning is indeed likely to berarer. However, Dycus assumes that learners must guess the exact meaning. Returning
to our earlier example, the words crippled, destroyed, devastated and ravaged would all have
similar meaning in the provided context, only diﬀering in slight and subtle ways. On ﬁrst
encounter, a learner would not be able to determine these nuances, but could still narrow
down the word’s potential meaning to some broad sense shared between these words, anabstract concept of damage and destruction. Subsequent encounters or some additionalsource of information are needed for learners to acquire the new word’s full meaning. IFC
remains useful as a technique for gradually reﬁning a new word’s meaning until the learner’s
understanding approximates that of the L1 population.
In addition to IFC or as an alternative to IFC, unknown words can be ignored or they
can be looked up in a dictionary. If ignored, the learner suﬀers the resultant penalty tocomprehension. If a dictionary is used, the learner suﬀers from the costly time-delay before
meaning is provided. e longer the delay, the less likely the learner is to continue to retain
the earlier context in their working memory, making swift dictionary lookup important forcomprehension. In our earlier example, dictionary lookup of crippled after guessing would
also have allowed the learner to pick up diﬀerences between it and its near-synonyms.
In advocacy of IFC, Fraser’s (1999) study showed that inferring meaning gave similar
vocabulary retention to dictionary lookup, although accuracy was lower. e highest retention strategy was to guess the word’s meaning, then conﬁrm the meaning by dictionary
lookup; the deeper processing of the combined approach explains the higher retention rate.
Mondria (2003) performed later experiments where learners were given pregnant contexts
so that inferring an incorrect meaning would be unlikely, and found instead that inferring

Chapter 3: Learning Japanese vocabulary

35

and lookup had similar retention rates, despite the use of a veriﬁcation step in the IFC
strategy. It may be that semantic engagement is increased by the process of reconciling an
incorrect guess with an authoritative word deﬁnition in the veriﬁcation step.
e partial successes of the guessing strategy could be explained by its eﬀectivenesswhere there is a large amount of beneﬁcial L1 transfer, for example, between languages
that share Latin as a common ancestor. In Japanese many loanwords are borrowed fromEnglish or other European languages and written in katakana. Guessing the meaning of aloanword may be eﬀective when the source word is in the learner’s L1, and phonetic andsemantically similarity to its source word overlap. However, in many cases the strategy will
still have diﬃculty: the source word could be from another language; the nearest phonetic
neighbour could be a diﬀerent word; the loanword or the source word could have drifted inmeaning from the time when the loan originated; or, the word could be an foreign loanword
construction not used outside Japan.6
Suppose that no beneﬁcial L1 transfer is available to aid IFC. en the eﬀectivenessof IFC is supported by a compositional ordering strategy for learning words, since such a
strategy will allow learners to utilise the internal cues given by a word’s components as wellas its context in order to infer its meaning. More broadly though, we take the position thatmany of the diﬃculties in applying IFC relate to the lack of an adequate base vocabulary,
relative to the text being read.
To summarise, learners are typically unable to use guessing eﬀectively until they reachvery large vocabulary sizes, except in the presence of strong, beneﬁcial L1 transfer. eythus rely on dictionary lookup heavily to aid them through their reading until they reach alevel where they can utilise this technique usefully. Any improvements we can thus maketo dictionary lookup will save much time for these learners, and may also help to maintainmotivation to read.

For example マイナスドライバー mainasudoraibā “ﬂathead screwdriver”, from “minus (screw)driver”.
Such constructions are known as 和製英語 wasei eigo “Japanese-made English”.
6

36

Chapter 3: Learning Japanese vocabularySelf-study tools
A vast number of vocabulary and kanji study tools are available on the web for Japanese,perhaps more so than for other languages. Much of this richness is attributable to the availability of the EDICT Japanese-English dictionary7 constructed by Jim Breen in 1991, andthe matching KANJIDIC dictionary of kanji meanings.8 ese resources allowed developers of study tools and dictionary interfaces to focus on user interaction rather than on
lexicography. For a broader selection of study tools than we can discuss here, we refer theinterested reader to Jim Breen’s Japanese Page9 and a recent portal built for this purpose by
the Japan Foundation called Nihongo-e-na (日本語eな).10 German speakers may also be
ﬁnd the Wadoku Wiki a useful resource for Japanese learning.11
Ultimately, each of tools in this section has at its core one of the two themes from theprevious section, either helping the user to memorise words or aiding them in acquiringthem through naturalistic input. We discuss both forms of self-study tool in this section.
Memorisation tools
e ﬁrst tool we examine is Quizlet,12 a language independent ﬂashcard tool for memorisation of vocabulary or even general facts. Whilst there are many ﬂashcard interfaces
online, most are highly specialised and very limited in scope. Quizlet is interesting for several reasons. It allows users to develop ﬂashcards themselves from their own study materials,
to share their ﬂashcard sets with others, and to study a variety of diﬀerent modes. Its mainmode is a learning mode, where the learner must type explicitly the answer to the ﬂashcard,or else postpone answering the question until later in the session. Each session thus constitutes one successful pass through the ﬂashcard set. It also provides a test mode, which cangenerate tests from the ﬂashcard set using multiple-choice or productive question formats,and two game modes for some variety.
Despite its usefulness as a general learning tool, Quizlet has two main limitations.
7

http://www.csse.monash.edu.au/~jwb/j_edict.html
http://www.csse.monash.edu.au/~jwb/kanjidic.html
9
http://www.csse.monash.edu.au/~jwb/japanese.html
10
http://nihongo-e-na.com/
11
http://www.wadoku.de/wiki/display/WAD/WadokuWiki
12
http://quizlet.com/
8Chapter 3: Learning Japanese vocabulary37

Figure 3.2: e basic study mode of Quizlet, shown part way through study of the JLPT 4kanji set. e JLPT 4 kanji list was entered by another user.

Firstly, we know that word knowledge is multi-dimensional, yet the general-purpose natureof its ﬂashcards limit users to studying a single dimension of knowledge at a time. is criticism applies to the ﬂashcard paradigm as a whole, although other more specialised ﬂashcard
sets – for example the paper ﬂashcards developed by White Rabbit Press for kanji study13 –oﬀer far richer linguistic aids which might help learners to acquire broader aspects of word
knowledge. Secondly, for all its basis in memorisation Quizlet doesn’t provide support forspaced repetition in any form, thus undermining its usefulness in developing long-term
retention.
We discussed earlier the Heisig method as advocating a particular form of mnemonic,
where a story is constructed for a kanji involving either aspects of its form or its component
radicals. An innovative site supporting this study method is the Reviewing e Kanji site.14
Heisig (1985) advocates learners developing their own mnemonic stories, however this cansometimes be diﬃcult if the elements a learner must include in these stories seem unrelated.
13
14

http://www.whiterabbitpress.com/catalog/Flashcards-orderby0-p-1-c-248.html
http://kanji.koohii.com/

38

Chapter 3: Learning Japanese vocabulary

Figure 3.3: e “Scatter” game in Quizlet, where users must drag and drop matching elements of word-gloss pairs onto each other to eliminate them. e user is scored by theirtime taken, which is compared to their previous best.

Chapter 3: Learning Japanese vocabulary

39

Reviewing the Kanji allows learners to share mnemonic stories and vote on them, thus
providing a very useful aid to this form of memorisation.
Reviewing the Kanji uses a form of spaced repetition developed by Leitner (1972) anddiscussed later by Mondria and Vries (1994), where a number of bins are used to store thediﬀerent concepts being memorised (Figure 3.4). Each bin is used in a self-test, where a
successful recall causes the concept to advance to the next bin, and a failed recall moves theconcept back to the starting bin. Each successive bin represents an increasing timeframe be-

hZkZgVa ldgYh d[ V hZ

fore the next self-test: for example, the succession 1 day, 3 days, 1 week, 1 month, 6 hZchZidejii]ZhZldg
monthsI]Z]VcYXdbejiZg^hVYZX`d[XVgYhl^i]Vhde]^hi^XViZYgZeZi^i^dchnhiZb#I]^h
ZmVb^cZ l]Vi i]Z
could be used.
is system reduces the amount of time spent on successfully learnedidwords,
YZX`d[XVgYh·^c^ihh^beaZhi[dgbVh]dZWdm·^hY^k^YZY^cidÄkZXdbeVgibZcihd[ i]Zb ZmVXian! VcYd[^cXgZVh^c\h^oZl^i]Vl^Yi]d[&!'!*!-VcY&)XbgZheZXi^kZan#6cdi]Zgedh"and yet also
revisits them in suﬃcient spacing to promote their long-term accessibility
id VYYincZl ldgYh dg
h^W^a^in^hiV`^c\VhZg^Zhd[ÄkZVjY^d"XVhhZiiZWdmZh!Wji^ci]ViXVhZi]ZgZ^hcd^c"memory. XgZVhZYh^oZd[i]ZXdbeVgibZcih#I]ZVYkVciV\Z!]dlZkZg!d[i]^hh]VeZ^hi]Vii]Z

bZVc^c\#bViZg^VaXVcZVh^anWZXVgg^ZYVgdjcY!VcYXVcWZegVXi^XZYViVcnheVgZbdbZci#

Coj^*0ºMehZii^eb[Whd[Z_dYedj[nj$»I]ZgZ^ha^iiaZYdjWiVW
cZhhd[XdciZmihhjX]
ViZmi^ci]ZegdXZhhd[6XdciZmicdidcanh]d^ihjhZ!Wji^iXVcVahd]Z
ldgYVcY^ihbZVc^c\;dgZmVbeaZ!hdbZdc;gZcX]ldgYlIgV_ZXidgnd[V`cdlcldgYi]ZhZciZcXZ
IgV_ZXidgnd[VldgYcdi`cdlcdgcdadc\Zg`cdlc
i]ZaZVgcZgYdZhcdig=dlYdZhdcZldg`l^i]i]Z]VcYXdbejiZg4:kZgnldgYidWZaZVgcZY^hejidcV
Figure 3.4:XVgY/i]Z[dgZ^\c"aVc\jV\ZldgYdcdcZh^YZ!i]ZigVchaVi^dcdci]Zdi]ZgegZ[ZgVWane card system proposed by Leitner for spaced repetition. From Mondriai]ZldgYbZVch(2007). ^cVY^[[ZgZciXdadjg!hdi]Vi^i^h^bbZY^ViZanXaZVgl]^X]^hi]Z[gdciVcYl]^X]^h
gZbZbWZgi]Vii]Zldi]ZWVX`#I]ZXVgYhVgZeaVXZY^cXdbeVgibZci&(%)%XVgYhVii]Zbdhi!V[iZgV hZciZcXZ l^i]
l]^X]i]ZnVgZegVXi^XZY#I]ZldgYhi]ZaZVgcZg`cdlhVgZbdkZYdcidXdbeVgi"! l]^X] gZb^cbZci'0i]ZdcZhi]ZaZVgcZgYdZhcdi`cdlVgZeji^ci]ZWVX`d[XdbeVgibZci&d[i]ZbZVc^c\d[V\V^c#L]Zci]ZgZVgZdcanV[ZlldgYhaZ[i^cXdbeVgibZci&!i]ZXdbeVgibZci^h
iZmiXVc]Zae^cgZiV^cgZeaZc^h]ZYl^i](%)%cZlldgYh!VcYldgYhXVcWZaZVgcZYVcYbdkZYdcV\V^c#
Reading aidsi]ZgZ[dgZ ^i hZZbh ad
6[iZgi]^h]VhWZZcgZeZViZYV[Zli^bZh!XdbeVgibZci'WZXdbZhÄaaZYje!VcYi]Z
i]VildgYhh]djaYVal
i^bZ^hg^eZidgZeZVii]ZldgYh^cXdbeVgibZci'/i]ZdcZhi]ZaZVgcZghi^aa`cdlh^cXdciZmi#
In contrast
toﬂashcard
tools,
thelast
twosystems
weconsideraim
topromotesuccess\d^cidXdbeVgibZci(0i]ZdcZhi]ViVgZ^bbZY^ViZangZXVaaZY\dWVX`^cidi]ZWVX`=dlZkZg!i]ZgZVgZild[XdbeVgibZci&#>ci]^hlVnVWdjii]ZWgZVYi]d[VÄc\Zg^hXaZVgZYdjid[Xdb"
in readingeVgibZci'#I]^h^hVahdi]Zeg^cX^eaZd[i]Z]VcYXdbejiZg^cVcjih]Zaa/l]ViZkZg
by augmenting computer-based texts with dictionary glosses. is approach»gjaZd[i]jbW¼#;^ghi!b^h»`cdlc¼\dZh^cidi]ZcZmiXdbeVgibZci0Vcni]^c\»cdi`cdlc$cdadc\Zg`cdlc¼
ldgYhXVcWZaZVgcZYZ
is comparable to preparing a graded reader where rare or out-of-syllabus words arepreh]djaY\dWVX`^cidXdbeVgibZci&#dji XdciZmi# EgZhZci^
15
glossed, but
preparing text in this way using a manual annotation system such as JGloss
6[iZghdbZi^bZXdbeVgibZcih(VcY)l^aaWZXdbZÄaaZYje#>ci]ViXVhZi]ZhVbZ
l^i]djiVXdciZmi·[dgegdXZYjgZ^hVYdeiZYVh^cXdbeVgi¬bZci'#L]ZcZkZcijVaanXdbeVgibZci*VahdaZVgcZgVh`h[dgi]Zb·15
WZXdbZhÄaaZYjeVcYgddb^hidWZbVYZi]ZgZidd!i]ZaZVgcZgbVn^ceg^cX^eaZ
http://jgloss.sourceforge.net/
XVabZi]dYi]ViegZkZi]gdli]ZXVgYhVlVn^[]Zdgh]Zhi^aa`cdlhi]Zb/[dgWni]Zci]ZldgYh]VkZWZZc[gdb]Vk^c\id^ckZc
hZZchdd[iZc!VcYhdbjX]Y^hig^WjiZY^ci^bZVii]Vi!i]VilZbVnheZV`d[gZVadgjhZ[jaXdciZmi!l]^X`cdlaZY\Z#6cVaiZgcVi^kZedhh^W^a^in^hidldg`l^i]VXdbeVgibZci*i]Vi^hhd]VhidWZlg^iiZcYdlc
W^\i]Vi^icZZYcdiWZXaZVgZYdji#>ci]^hXVhZi]ZXVgYhbVnWZ`ZeiVcYdXXVh^dc"
VcYhddc!ZVX]i^bZ#
Vaani]ZaZVgcZgXVcYgVlVhVbeaZd[iZcXVgYh[gdb^i^cdgYZgidhZZ^[i]ZldgYh
HZXdcY!VcYi]^h^hVX
VgZhi^aa`cdlc#>ci]^hlVn!XdbeVgibZci*XdciV^chi]ZaZVgcZg¼hXdbeaZiZldgY
ed^ci!aZVgc^c\VldgY`cdlaZY\Z#
XdciZmibVngZhjai^c^c\i]ZldgYdcan^cldghZ/cdiZkZcgZXd\

40

Chapter 3: Learning Japanese vocabulary

is time consuming. Instead, the PopJisyo16 and Rikai17 sites both provide a manner of
pre-processing an arbitrary Japanese text, and loading it with pop-up translations for all
recognised words. Since the translations only pop-up if the user places their mouse over theword, they are not obtrusive and distracting for readers who already know a word’s meaning.
However, when a word is unknown, the eﬀect is that of instantaneous dictionary lookup.
Figure 3.5 shows such an example where Rikai is used to aid reading of a Japanese news
article. A similar system is provided by the the Reading Tutor toolbox,18 which prepareson-demand translations in a right-hand column next to the text.

Figure 3.5: Rikai used to aid reading of a news article from Asahi Shimbun. When themouse is hovered over the word 商会, its gloss “ﬁrm; company” is displayed, as well as anindividual gloss for each kanji in the word.
For all of these systems, the lack of a signiﬁcant time-cost penalty for unknown words
could have several undesirable consequences, in particular over-reliance on and abuse of16

http://www.popjisyo.com/WebHint/Portal_e.aspx
http://www.rikai.com/perl/Home.pl18
http://language.tiu.ac.jp/index_e.html
17Chapter 3: Learning Japanese vocabulary

41

the dictionary as a learning tool. Critics of paper-based dictionaries charge that excessive
dictionary use leads to reliance on word-by-word translation, potentially ignoring multiword
expressions, idioms and issues of lexical gridding (Walz 1990; Bell 1998; Prichard 2008).
Furthermore, with instantaneous lookup the incentive to learn a word may be too small
for a learner to invest the time and attention required to learn it. Nonetheless, they allow
learners of a much greater variety of proﬁciencies to read authentic texts of their choosing,and are thus potentially a strong aid in facilitating the virtuous circle of successful reading.
A second criticism is that they provide little support for reading-for-vocabulary, in thesense that most learners will ﬁnd too many new words to keep track of, with little to diﬀerentiate which new words should be further studied. Learning a new word takes time, but
the payoﬀ in doing so varies according to the word’s frequency of occurrence. Of the threesystems only Reading Tutor provides cues as to frequency, by colouring words in the source
text by their diﬃculty according to Japanese Language Proﬁciency Test ( JLPT) levels, andnone of the systems provides a more integrated user-oriented lexicon based on actual exposure to words.
Finally, these systems are limited by the media the learner is trying to read, and oﬄine
or scanned texts without optical character recognition are still beyond the reach of thesereading aids. For such media, the learner must consult a dictionary system.

3.2

Structure of the mental lexicon

Having made our case for supporting autonomous study of vocabulary, and having discussed how words are chosen for study and then learned, we now consider how we might
improve the learning process. We contend that if we more closely model how learners makeerrors, particularly during reading, we can vastly improve the tools which support their vocabulary acquisition. For this reason, we now consider contemporary models of the mental
lexicon.
Such models are important for two main reasons. Firstly, they are suggestive of how tobest populate the lexicon with new L2 words, which is indeed the focus of this thesis. Secondly, the relationships between words, especially those which emerge due to the lexicon’s
structure, might be leveraged to support vocabulary acquisition in a variety of applications.

42

Chapter 3: Learning Japanese vocabulary
As writing systems vary signiﬁcantly as to what information they encode (Koda 2007),

our investigations of graphemic similarity will inevitably be writing-system speciﬁc. In
our discussion, we try to maintain a multilingual perspective where possible, focusing on
Japanese only where necessary.

Visual word recognition
We learn about the structure of the mental lexicon primarily through studying visual
word recognition, with the caveat that we are considering the ultimate structure for skilled
L1 readers. Models of visual word recognition have to account for many known psycholinguistic eﬀects if they are to be successful (Taft 1991:11; Handke 1995:166). Of these,
Lupker (2005) suggests that the most important are the word superiority, word frequency,
semantic priming and masked repetition priming eﬀects.
In general, proposed models fall into one of two categories: search models where someautonomous search process identiﬁes candidates from sub-lexical features and proceeds toverify them until a match is found, or activation models where continuous activation occurs
between various levels of processing, and the word perceived is that with the strongest activation level (Lupker 2005). In this section we focus on two variants of the basic interactiveactivation model, which we show in Figure 3.6, and which address each of these key eﬀects
appropriately.
Note that we focus on the word level in isolation, rather than considering larger units
such as phrases or sentences. As a receptive model, we model the process initiated by visual
input and concluded by successful access to a word. is can be compared with productivemodels, for example that provided by Levelt et al. (1999) for speech production. oughnot shown on Figure 3.6, variants of the interactive-activation model posit an intermediatelevel between the letter and word levels which activates related phonological units. If we areto consider Chinese or Japanese characters, then radicals seem to be appropriate sub-word
units to occupy this level.
Saito et al. (1998) adapted this model to Japanese, developing the Companion Activation Model (CAM) shown in Figure 3.7. e CAM does indeed use radicals as intermediate sub-word units, however above radicals and below words lies the additional whole-Chapter 3: Learning Japanese vocabulary43

Word level

Letter level

Key:

Activation
Inhibition

Feature level

Visual input

Figure 3.6: e multilevel interactive-activation model for alphabetic orthographies. From
McClelland and Rumelhart (1981).

44

Chapter 3: Learning Japanese vocabulary

kanji level. ey also assume two types of activation (foreground and background) whichinteract with one another. When a kanji radical is activated, in combination with otherforegrounded radicals it activates whole-character matches in the foreground, and visualneighbours which also contain the radical (“companions”) in the background. eir results
in several experiments are interpreted to mean that phonetic properties of radicals are automatically activated. ismatches the results of earlier experiments by Flores
FORM AND SOUND SIMILARITY EFFECTS IN KANJI RECOGNITION
329 d’Arcais et al.
(1995), where pre-exposure to a phonetic radical reduced latency in a naming task.

Figure 1. A schematic illustration of companion-activation model (CAM) (after, Saito 1997).

Figure 3.7: e Companion Activation Model (CAM). From Saito et al. (1998).
[ 175 ]

Chapter 3: Learning Japanese vocabulary45

Although Saito et al.’s (1998) model only considers kanji split into left and right components – roughly 53% of the JIS standard set – it could easily be extended to other kanji
shapes: only left-right kanji have phonetic radicals with any reliability, so for other shapes
we would assume that radical-level phonology is not activated. It also provide explicit suggestions of other candidates which are considered in the recognition process, thus implicitly suggesting potential error outcomes should misrecognition occur. Unfortunately, theCAM provides little explanation of the role of multiple scripts in word recognition. It also
suﬀers from problems of “representational redundancy, homographs, and varying degrees
of semantic transparency” (Joyce 2002:80). Motivated by these problems, an alternativeinteractive-activation model was constructed by Taft et al. (1999a), and adapted for Japanese
JPR009.fm Page 89 Wednesday, March 13, 2002 6:32 PM

by Joyce (1999).

Constituent-morpheme priming
89© Japanese Psychological Association 2002.

Figure 1. Lemma unit connections in a multilevel interactive-activation model for Japanese.

Figure 3.8: Lemma unit connections in Joyce’s (2002) multilevel interactive-activation
model for Japanese.
is model is very similar to CAM in its broad structure, in that activation ﬂows from thelower levels upwards through to meaning, however lemma units are added which mediate
between orthography, meaning and phonology (Figure 3.8). ese abstract units roughly
represent the morpheme level, and they contain additional information about their links toorthographic and phonetic units which allows them to order their connections. Amongst

46

Chapter 3: Learning Japanese vocabulary
Link type
Semantic

Intralingual(near-)synonymy/antonymy
hyponymy
meronymyholonymy
entailment

Interlingual(near-)synonymy
cognatePhonetic(near-)homophony(near-)homophony

Graphemic(near-)homography(near-)homography

Table 3.3: Types of lexical relationships available to the ideal bilingual speaker.
its other advantages, this model provides clearer integration of the multiple scripts used inJapanese.
Each of these models makes heavy use of the hierarchical nature of kanji, which is unsurprising, given that such structure exists. Activated candidates during the reading process
are suggestive of potential erroneous outcomes if a word is misread. ey also give us insight into how words may become related through the recognition process. For example,
kanji sharing radicals may be commonly activated as candidates when processed throughreading, and this common activation may develop a bond between these characters. We
now consider these and other relationships which form between words.

General lexical relationships
Our discussion of Japanese-speciﬁc aspects of the mental lexicon was dominated by theprocessing of kanji, and kanji compounds. In this section we consider more broadly the
relationships between words which occur within any language, and indeed interlingually
between words of diﬀerent languages. Ultimately, quite a large inventory of such relationships can be uncovered, as shown in Table 3.3.
As is clear from Table 3.3, the richest variety of relationships is semantic and available
monolingually, i.e. between words of the same language. For semantic relationships within
English the primary resource is WordNet, a semantic network supporting several relations(Fellbaum 1998), following earlier work on lexical relations by Cruse (1986). Firstly, words
are grouped into synonym sets (“synsets”). For nouns, hyponymy (the is-a/kind-of relation-

Chapter 3: Learning Japanese vocabulary

47ship) and meronymy (the part-of relationship) are provided; for verbs, hyponymy, troponymy(the manner-of-doing relationship) and entailment are available. For Japanese, the GoiTaikei project (Ikehara et al. 1997) also provides a hierarchical semantic lexicon, using a
subset of the relations available for the WordNet project. More recently, a Japanese version
WordNet was constructed using the synsets from the English WordNet as its base (Isahara
et al. 2008).
We refer to semantic links in general as associations, and our listing of association types
should not be considered exhaustive. ese rich semantic relations are still insuﬃcient tocapture the semantic proximity of pairs such as doctor and patient. For this reason, work on
circumventing the tip-of-the-tongue problem (Zock 2002; Zock and Bilac 2004) has more
recently focused on extending WordNet with syntagmatic relationships (Ferret and Zock2006). Other relationships may also exist: for example, recent experiments by Gaume et al.
(2008) comparing child and adult learners of French found a salient semantic relationshipbetween verbs based on co-hyponymy. Overall, suﬃcient proﬁciency in any language yields
associations between words which are shared across speakers, and these can be captured bystudying suﬃcient numbers of speakers, as in the case of Joyce’s (2005) large-scale databaseof such associations for Japanese.19
If we now consider interlingual word relationships the most obvious is synonymy, or
more accurately near-synonymy. Every bilingual dictionary attempts to match words in
one language with their closest synonyms in another. If we consider some abstract spaceof all possible meanings, then each language partitions and covers this space with words ina diﬀerent manner. is has been called lexical gridding, with the idea that the “grids” of
two languages often do not “line up” (Laufer-Dvorkin 1991:16). For example, in English
we describe the middle front of a creature’s head as either its nose (for people), muzzle (fordogs), snout (for pigs), or trunk (for elephants); in Japanese, the single word 鼻 hana isused in all these contexts. In more complex examples, words only partially correspond to
one-another by overlapping in a few of their available senses, but not in others. Ploux andJi (2003) visualise this situation by constructing two-dimensional semantic maps of nearsynonymous English and French words. Lexical gridding eﬀects in general are not limited19

http://www.valdes.titech.ac.jp/~terry/jwad.html

48

Chapter 3: Learning Japanese vocabulary

to denotational meaning, but cover all aspects of word knowledge, as we shall later discuss(Section 3.1).
When L2 words sound like words from our L1, or are written like words in our L1, we
associate them with these words, regardless of semantic diﬀerences. Indeed, contemporary
product names are chosen carefully to avoid negative associations with consumers from
diﬀerent linguistic backgrounds. When two languages share some common ancestry, wordpairs will exist between these languages which share the same etymology. ese word pairs
are cognates, and due to the slow speed of linguistic change, these pairs are often nearsynonyms, near-homographs, and near-homophones.
/#dres/ address/d$kst$r%ti/ dexterity

adresse /adr$s/
dextérité /d$ksterite/“telephone” /deNwa/ 電話#$!/diànhuà/ “telephone”“letter” /tegami/ 手紙!!/zh!/ “letter”"!!/sh"uzh!/ “toilet paper”

!

Key:

Meaning
Orthography
Sound

Figure 3.9: Interlingual relationships based on near- homophony, homography and synonymy. e ﬁrst example is English-French; the second and third are Japanese-Chinese.
We can bring together many of these ideas with the examples shown in Figure 3.9. In
the ﬁrst example, the French word adresse is cognate with the English address, and shares themeaning “postal address”. However, it also means “dexterity”, which itself has a matching
cognate dextérité in French. Both cognate pairs show correspondence in meaning in atleast one sense of each word. Importantly, the cognate relationship is predictable fromthe orthographic and phonetic similarity. In our second example, a similar situation exists
between compounds in Chinese and Japanese; again the cognate relationship is predictable
and reliable. In our ﬁnal example, the cognate relationship between 手紙 and 手纸 exists
between characters at the morpheme level, but not at the whole-word level. e subsequentmismatch in whole-word meaning is signiﬁcant, and in such cases we call the pair false
friends, indicating the problems they cause for learners.
Altogether, these relationships transform our view of the mental lexicon from a series ofChapter 3: Learning Japanese vocabulary49

isolated words to a densely interconnected graph. Lexical resources attempt to represent a
subset of the vertices and edges in this graph. In Japanese and Chinese, their rich orthographies provide substantial graphemic relationships based on near-homography which have
not yet been fully explored. ese relationships could provide some important structure forlearners, and deserve more attention.

Graphemic neighbourhoods: errors and associations
If we measure the physical similarity of two written symbols, for example using metricmeasurements or on-screen pixels, we expect that the closer the two symbols the more
likely they are to be confused. is is trivially true at extreme levels of proximity, but how
the confusability changes as the distance between symbols increases depends strongly on
the psychological reality of how the symbols are perceived. It is precisely this reason we
looked at the word recognition process for Japanese earlier in this section. e generalvisual neighbourhood around a Japanese character – presumed to be shared between native
speakers – remains unknown, though several experiments give useful indicators as to thewriting system’s topology.
e concept of a visual neighbourhood is important because a word’s neighbours are
assumed to be competing candidates during the word recognition process, precisely because
they share salient visual features with the target word. Applications wishing to predictvisual recognition errors should thus focus on visual neighbours as error candidates. Our
previous discussion considered what lexical relationships might be consciously available toreaders, but these conscious graphemic associations may be aﬀected by visual recognition
processes. We ﬁrstly discuss these processes, before considering evidence for the topologyof such neighbourhoods.
Accessibility of graphemic neighbours
L2 learners and bilingual speakers clearly associate visually similar words interlingually,usually as cognates or mnemonic aids. However, the extent to which these high-similarity
graphemic relationships are readily accessible within a single language is unclear becauseof conﬂicting theoretical predictions from lexical competition and global activation eﬀects.

50

Chapter 3: Learning Japanese vocabulary

Lexical competition suggests that in processing the stimulus word, visual neighbours of thatword will be inhibited (Figure 3.6). In contrast, global activation suggests that the increased
activation in that region will increase the activation in neighbours of the stimulus as well.
In fact, evidence from Carreiras et al. (1997) suggests that both can occur and are taskdependent: orthographic density about a word facilitated naming and lexical decision tasks,
but had an inhibitory eﬀect in a progressive demasking task.20
If the stimulus word is previously unseen by the reader, then several possibilities areraised. Lexical retrieval will fail, so we might expect inhibitory eﬀects at the word-level toreduce or disappear, thus making graphemic neighbours more consciously accessible. On
the other hand, Gaskell and Dumay’s (2003) study of spoken word recognition suggested
that the ﬁrst few exposures to non-words can activate the nearest known word. is result
seems intuitive: for example, if a new word epple was coined in English and presented as
stimulus, we might expect readers to mentally access its neighbour apple. is could occur
regardless of whether they thought the new word was a spelling mistake or not. If thiseﬀect also occurs in visual word recognition, inhibition eﬀects could occur from the closest
neighbour. However, we cannot examine these eﬀects more carefully without deﬁning ournotion of neighbour more carefully.
Graphemic neighbourhood topology
Studies of visual word processing in alphabetic orthographies have long been interested
in how the behaviour of such processing changes in response to many variables, including
visual neighbourhood density. For this reason, Landauer and Streeter (1973) introduced
the N metric of visual density around a word. N is calculated as the number of words which
diﬀer from the original in one character only, implicitly deﬁning this as the neighbour
criterion. However, the many applications for orthographic similarity – including opticalcharacter recognition (Taghva et al. 1994), record matching (Cohen et al. 2003), spellingcorrection (Wagner and Fischer 1974) and others – have developed far richer string distance
20

Progressive demasking is a variant of the naming task where a word (e.g. “ladder”) is alternated witha visual mask (e.g. “######”) on-screen. First the mask is displayed, then it is replace very brieﬂy by theword. e mask and word alternate, and in each cycle the mask reduces in duration and the word increases induration. e participant must read the word out aloud as soon as they can identify it, and the time taken forthem to do this is recorded.

Chapter 3: Learning Japanese vocabulary

51

metrics than this. ese richer metrics potentially provide the means to model much moreclosely the psychological reality of word perception, however the expense of human trials
and lack of data on human similarity perception makes their evaluation diﬃcult. Furthermore, choosing between them on a theoretical basis is diﬃcult when many may use featuresof unknown cognitive salience.
e case for Japanese and Chinese is similar, in that the psychological literature deﬁnes
only a basic orthographic neighbourhood relationship, where neighbours of a kanji may
have one radical from the original kanji swapped, or identical radicals but diﬀerent layouts(see Saito et al. (1998) or Taft et al. (1999b)). However, the space of broader kanji distancemetrics remains impoverished in Japanese for two reasons. Firstly, many applications ofsuch metrics in English have no equivalents in Japanese. For example, computer input ismediated by an IME and the types of mistakes made do not equate directly to spelling errorsthat could use such distance metrics for correction.21 Secondly, unlike words in alphabetic
orthographies whose layout is constructed linearly of symbols, kanji have a complex, nested
two-dimensional layout, limiting the ability of abstract string distance metrics to transfer
to the kanji distance problem.
So how might we best go about measuring and predicting the psychological reality ofkanji similarity? Visual search tasks, such as the one described by Yeh and Li (2002) for
Chinese, provide a good start. In the task, native Chinese speakers had to determine if atarget character was present amongst distractors. Yeh and Li found that shared structure
between target and distractors provided the strongest interference, slowing decision times.
ey investigated two types of characters: those split into left and right components, andthose split into top and bottom components. Shared radicals also slowed decision times, but
only where the broad layout was also shared. is suggests that the layout of a characters into
broad components is dominant initial feature used in kanji perception, and that components
are considered as secondary features. is makes intuitive sense, since most kanji are coveredby a handful of structural variants. For example, the popular SKIP indexing scheme uses
a choice between four basic structures as its primary index feature (Halpern 1999). We
discuss such indexing schemes in detail in Section 3.3 which follows.
21

Common mistakes when using an IME include incorrectly choosing the ﬁrst kanji compound displayedin the IME, when the correct compound appears later in the list.

52

Chapter 3: Learning Japanese vocabulary
Despite this high-level information on neighbourhood topology, how these neighbour-hoods are structured remains open. Furthermore, applications cannot make use of thisinformation until at least some of this structure is resolved.

3.3

Dictionary lookup of Japanese

Lookup for producers
Producers of language – speakers or writers – begin with a meaning in mind, and arelooking for the right words to use to express that meaning. Even in a monolingual context,
ﬁnding the right word can be diﬃcult, whether the problem is one of limited knowledge orsimply one of access, as in the case of the tip-of-the-tongue problem.
Access based on meaning (onomasiological search) requires ﬁrstly expressing that meaning somehow. is is done almost exclusively through other semantically proximate or related words. We discussed types of semantic relationship earlier in Section 3.2; each ofthese relationships deﬁnes a method of semantic access for producers, and indeed resources
providing these relationships form lookup methods themselves. In monolingual context,
the user speciﬁes the meaning by using a near-synonym; this leads to the traditional supporting resource, a monolingual thesaurus such as Roget’s International esaurus (Rogetand Chapman 1977). In a multilingual context, the learner can specify the meaning in their
L1; the traditional resource is thus an L1-to-L2 dictionary, making available the same synonymy relationship but between bilingual word-pairs. Occasionally these two resources are
merged, as in the “Hebrew-English-English” dictionary described by Laufer and LevitzkyAviad (2006).
e main exception in the dictionary space is the use of semagrams by Moerdijk et al.
(2008) in the Algemeen Nederlands Woordenboek, an online scholarly dictionary of contemporary standard Dutch currently under construction. A semagram is similar to a semantic frame, but serves as a formalised structure for the meaning of a word rather than foran event or state of being. Each word sense is a member of one or more semantic classes,
and each class has a type template deﬁning which slots are available and values these slots
can take. e term semagram refers to a type class with values. For example, the semagramChapter 3: Learning Japanese vocabulary53

for cow has slots for size, colour, build, function, and many other aspects of meaning, basedon its upper category as an animal and the type template for animals. is rich semantic
encoding for dictionary entries provides a correspondingly rich semantic search which canbe performed across many aspects of meaning.

Lookup for receivers
Receivers trying to decode a word into meaning rely on looking up either its orthographic form or its pronunciation in a dictionary. For languages which are orthographicallyshallow, these two are roughly interchangeable; a transcribed pronunciation in such a language will match the correct word spelling or form. For deep orthographies such as Japanesewith complex form-to-sound relationships, form and pronunciation are markedly diﬀerent
sources of information to draw on. In this section we focus on lookup for Japanese andChinese.
For these languages, lookup is also the mirror-image of the input problem. If the character or word can be input quickly and accurately, lookup is trivial. If a method allows
quick and accurate lookup, it likewise has potential uses as an input method. e problem
of input and that of lookup by form are thus equivalent. Where input methods may be circumvented, for example by copying and pasting a problematic word from an electronic text
into a dictionary, lookup is also trivial – hence the availability of the reading aids discussed
earlier. We also saw earlier that input by form is non-trivial, and discussed some solutions
from the input perspective. We now consider the lookup perspective of the same problem
by examining three diﬀerent kanji indexing schemes, as shown in Figure 3.10.
Traditional paper dictionaries contain three parts: an index of primary radicals (“section
headers”) ordered by stroke count, a per-primary radical index of characters which use it
indexed by remaining stroke count, and the dictionary entries themselves. Learners caneasily select the wrong component as the primary radical, or incorrectly count the numberof strokes, and thus have much trouble ﬁnding the character they seek. e multiple indexing steps are also time consuming, even without mistakes. e conversion to electronic
dictionaries eliminates the time it would take to turn pages in the paper dictionary, but stillrequires the visual scan of the original dictionary.

54

Chapter 3: Learning Japanese vocabulary

Traditional
1. IDENTIFY: 日 as section header
2. COUNT: strokes in 日, finding 4
3. LOOKUP: 日 in 4-stroke radical index
4. COUNT: remaining strokes, finding 4
5. LOOKUP: page no. for 明 in 8-stroke characters containing 日
6. LOOKUP: 明 at given page no.

明SKIP
1. IDENTIFY: shape as type 1 (left-right ⿰)
2. COUNT: strokes in 日, finding 4
3. COUNT: strokes in 月, finding 4
4. LOOKUP: 明 at index 1-4-4

Kansuke
1. COUNT: horizontal strokes, finding 6
2. COUNT: vertical strokes, finding 3
3. COUNT: other strokes, finding 1
4. LOOKUP: 明 at 6-3-1

Figure 3.10: Kanji lookup methods to ﬁnd 明 aka “bright”. Steps beginning with
“IDENTIFY” or “COUNT” are potential error sites for learners. Steps beginning with
“LOOKUP” involve use of an index, and subsequent visual scan for the desired item.

Chapter 3: Learning Japanese vocabulary55

e SKIP (System of Kanji Indexing by Patterns) system of lookup provides an alternative indexing scheme based on a kanji’s overall shape (Halpern 1999). Each kanji is indexed
by a three digit code. e ﬁrst digit represents its broad shape: ⿰ left-right, ⿱ top-bottom,
⿴ exterior-interior or ⿻ other. e remaining two digits are given by the stroke count of

the two separated components, except in the “other” case, where they specify one of four
sub-patterns and the overall stroke count. For example, 明 aka “bright” has skip code 14-4. e SKIP system has two main advantages. Firstly, it removes the need to identify aprimary radical for the character. Secondly, it uses only one index, ordered on a code which
can be determined without consulting the dictionary. is makes correct access about as
fast as alphabetic dictionaries for other languages, though mistakes in stroke counting canstill impede access.
e Kansuke electronic dictionary by Tanaka-Ishii and Godon (2006) aims to simplify
the stroke counting method in order to avoid such problems. Users instead form a threenumber code representing the number of horizontal, vertical and other strokes that makeup a character. To avoid stroke count ambiguity, complex strokes are split into their component parts. For example, the single hand written stroke ㇆ is counted as two strokes
when determining the Kansuke code, one horizontal and one vertical. Characters can also
be looked up from their components. For our earlier example, 明 consists of 日 with code
3-2-0 and 月 with code 3-1-1.
e Japanese-German dictionary constructed by Hans-Jörg Bibiko22 provides, amongst
other lookup methods, a method based on selecting from a large table of components. If aradical such as 日 is selected from the table, a shortlist of matches is presented showing kanji
containing this radical, but the table of components is also updated so that incompatible
components (i.e. those which never co-occur with the current selection) are removed. By
propagating constraints this way, the user can more easily ﬁnd the kanji they are looking
for. Multi-paradigm dictionaries such as JEdict23 also combine several forms of lookup ina single interface.
Each of these dictionaries provides a complementary approach which can be added to
the learner’s arsenal. Looking again at Figure 3.10, we can see that in newer systems the
2223

http://lingweb.eva.mpg.de/kanji/
http://jedict.com/

56

Chapter 3: Learning Japanese vocabulary

learner generates a single index key for the kanji, then performs just one lookup using thiskey, thus saving time.
e trend across these systems is to reduce the amount of assumed or required knowledge
to perform lookup. However, this means that additional knowledge is not used, even when
it is available. A competing trend is to allow the user to specify partial knowledge wheneverpossible. is approach is taken by the FOKS dictionary system.

FOKS dictionary
We suggested earlier that if the pronunciation for a word is known, dictionary lookup
becomes simple and fast. What of the case where the user only has partial knowledge ofthe pronunciation? Such a case is common in Japanese, where the correct kanji reading iscontext-dependent. In such a case, the FOKS (Forgiving Online Kanji Search) system byBilac (2005) provides a useful lookup method.
Suppose, for example, a user wishes to ﬁnd the word 山車 “festival ﬂoat”, but is unsureof its pronunciation. FOKS allows them to guess the pronunciation based on readings theyknow for each character in other contexts. In this case, they might combine 山 yama “mountain” and 車 kuruma “car” and guess the word reading as yamakuruma. In this case, the correct
reading dashi cannot be guessed from the word’s parts, but our educated guess would leadthe user to the word, and provide access to both the correct reading and meaning.
FOKS uses an error model to correct for the most signiﬁcant recoverable errors made byusers in their dictionary queries. Incorrect choice of kanji reading accounts for roughly 80%
of user errors, according to post-hoc log analysis (Bilac et al. 2004), often in combination
with other error types. ree other error sources are also signiﬁcant: incorrect voicing,incorrect gemination, and incorrect vowel length. Voicing and gemination are changes tocharacter-level pronunciation that occur during word formation, as discussed in Section 2.3,
and which can be incorrectly applied by learners. e third source of error occurs whena learner mistakes a long vowel for a short vowel, or vice versa. Vowel length errors are
expected to be especially prevalent with learners from languages such as English which do
not maintain an equivalent vowel length distinction. Like many dictionaries, FOKS also
supports the use of wildcards in queries, which oﬀers some help for lookup of partiallyChapter 3: Learning Japanese vocabulary

57

known words.

3.4

Second language testingis section provides an overview of the current state-of-the-art in second language
vocabulary testing. It is divided into three main parts, considering the role of testing inlanguage study, general testing theory, and ﬁnally current testing methods and their limitations. ese limitations motivate our work on adaptive randomised testing.

e role of testinge broad purpose of testing is to make decisions (Murphy and Davidshofer 1998:2).
In the context of language learning, the information from tests allows: the appropriate
reward for students through grades; the optimal focus of future learning opportunities, forexample targeting them to a particular student’s needs; and evidence-based evaluation ofcourse structure and teaching methods based on their measured performance.
Any aspect of language that can be learned can be tested, but since our focus is on vocabulary study, a simple answer to the question, “What should we test?” could be all aspectsof word knowledge. However, to do this even for a handful of words is prohibitive, since thetestable aspects of knowledge for any individual word is large. is dilemma is a form of
the bandwidth-ﬁdelity dilemma discussed by Murphy and Davidshofer (1998:143) in thecontext of testing, and attributed to Shannon and Weaver (1949), where the amount of information conveyed is traded oﬀ against the accuracy with which it is conveyed. In general,
tests bias towards breadth because testing depth is quite hard, given that the connections
between words form a signiﬁcant part of word knowledge (Meara 2009:74). In other words,
knowing a word at depth means knowing its relationship to words around it, so testing aword at depth means testing these relationships too.
As this indicates, breadth and depth are now believed to be strongly intertwined, anda comparison of depth and breadth tests by Vermeer (2001) found strong correlation between them. is supports the validity of focusing on breadth in testing, for example in the

58

Chapter 3: Learning Japanese vocabulary

vocabulary levels tests (Nation 2001:412),24 provided that such tests are appropriately constructed. In the levels test, learners are tested on words sampled from a variety of frequency
bands. By observing tail-oﬀ from more frequent bands to less frequent bands, the learner’s
lexicon size can be estimated.
A related and important consideration is the distinction between receptive and productive aspects of word knowledge discussed earlier in Section 3.1. If nearly any aspect of word
knowledge can be composed of both receptive and productive parts, which should we test?
Many researchers believe the receptive/productive distinction to be related to depth of word
knowledge, and that receptive skills are a subset of productive skills. is view is supportedby the fact that subjects score signiﬁcantly higher on receptive tests than productive tests(Nation 2001), indicating they are easier. As a secondary eﬀect, subjects’ scores increase if
they are tested in the same manner that they are taught (i.e. productively or receptively).
However, productive vocabulary tests are notoriously diﬃcult to construct, since a question
might typically require the subject to produce a particular word, but will not reward themfor unexpected but correct alternatives (Meara 2009:34). Many early tests constructed thisway thus yielded poor reliability, with widely varying scores on retests.
A rethink of productive testing, in the form of the Lex30 test (Meara 2009:37),25 elicited
free word associations in order to obtain a representative sample of learner vocabulary knowledge. e resulting scores, based on the occurrence frequency of the words a learner provided, yield an index for productive vocabulary knowledge, although they do not as yet
estimate it directly. By avoiding the issue of requiring a particular answer to be producedby the learner, the reliability of the method is increased. However, it remains unable to testa particular word of interest, instead testing a learner’s productive vocabulary as a whole.
Finally, an experiment performed by Mondria (2007) found that studying receptively
and productively yielded equivalent receptive recall, but that productive study takes longer.
She concluded that productive study (and testing) should only take place if productive ability
was important.

24
25

http://www.lextutor.ca/tests/
http://www.lognostics.co.uk/tools/Lex30/index.htm

Chapter 3: Learning Japanese vocabulary

59

Testing theory
Modern testing theory aims to guide test design so as to improve the various quality
attributes of tests, and provides statistical models for constructing and analysing tests. issection provides a brief overview of recent changes in testing theory, and aims to highlight
three long running and related trends in testing: ﬁrstly, the trend from subjective to objective testing; secondly the trend from estimating “true” test scores to estimating participant
ability; and thirdly, the trend from pen-and-paper testing to computer-based testing. Fora more comprehensive overview of modern testing theory than this section allows, we refer
the reader to Hambleton and Jones (1993).
Two core concepts are central to any discussion of testing: validity and reliability. A
test has validity to the extent that it actually measures the underlying attribute intended.
For example, a written test might be a poor method of measuring oral language ability andmight thus have low validity. Validity can also refer to the logical validity of arguments(decisions) made on a basis of test results. Reliability on the other hand is the extent towhich each individual’s test score is consistent and repeatable – free from noise or error –and is measured over a sample of test participants.
e trend towards objective testing is motivated by the desire to improve the validity andreliability of tests. Subjective tests are tests where the outcome is aﬀected by participant,examiner and marker bias, and these biases in turn reduce test validity. Objective tests
are tests which are unaﬀected by these biases, and they strongly favour question formats
with only one correct answer, for example multiple-choice questions or simple scales, thus
avoiding the subjectivity of marking more open-ended question types. A side-eﬀect of thistrend in testing is that the human element is removed from marking tests, allowing tests tobe assessed automatically by computer.
A parallel trend has occurred in the statistical modelling of tests and test responses, ashift from modelling test scores to modelling participant ability directly. is is most evident in the gradual transition from Classical Testing eory (CTT) to Item Responseeory (IRT). Both of these theories provide diﬀerent and competing methods of formallyassessing reliability. Classical Test eory models a test score as composed of an individual’s“true score” over the test and an error term, which should be minimised. Item Response

60

Chapter 3: Learning Japanese vocabulary

eory instead models each individual question as contributing information about a participant’s ability. is information is usually described in the form of an item-characteristic
curve, the plot of the likelihood of answering correctly as a function of participant ability(Bull and McKenna 2004:81). Crucially, this per-question modelling allows constructionof large item banks which are drawn upon in designing tests to provide a desired level ofreliability in ability estimates. Together, the shift to modelling ability and calibrating individual questions raises the prospect of adaptive tests where each individual is shown test
items which are maximally relevant to determining their ability level. However, adaptive
tests were infeasible before the advent of computer-based testing, our ﬁnal trend under discussion.
Computer-based testing oﬀers many beneﬁts and disadvantages over traditional penand-paper testing. For example, provided that tests are objective, marking can be done
instantaneously by computer, thus providing direct feedback to participants. e cost ofadministering tests to large populations is also signiﬁcantly reduced. e disadvantages include the large set-up costs even when testing small populations, the logistics of computerresource availability and use during test administration, and medium-of-delivery eﬀects(Chalhoub–Deville and Deville 1999). However, computer-based testing in combination
with the large item-banks calibrated with IRT makes computer-adaptive testing feasible,
since personalised tests can be constructed on-the-ﬂy and in response to an individual’sestimated ability so far.
Adaptive tests are simply tests where the questions asked vary for each examinee. If
examinees are shown diﬀerent questions, then the use of the total number of questions
correctly answered as a measure of test performance becomes meaningless. Indeed, use of
IRT to model item diﬃculty is usually a prerequisite for this form of testing; in this caseeach response from the individual yields better information about their ability estimate,which in turn is used to determine which question would maximally reﬁne that estimate.
In this way a test can be made more reliable, shorter, or potentially both.
e main problem which computer-adaptive tests still suﬀer from is one of limited item
bank size. is means that even adaptive tests reduce in validity when individuals re-test
themselves, since participants will encounter the same questions as they did in earlier tests
with high probability. Within a limited domain, that of vocabulary testing, this thesis aims

Chapter 3: Learning Japanese vocabulary

61⊕ Active(retrieval of form)⊕ RecallTranslate the L1 word into
the correct L2 word

⊖ Recognition
Choose the correct L2translation for this L1 word
amongst distractors.

⊖ Passive(retrieval of meaning)

Translate the L2 word into
the correct L1 word

Choose the correct L1
translation for this L2 word
amongst distractors.

Table 3.4: Four types of question used by Laufer and Goldstein (2004) in CATSS, eachindicating the task the learner must perform. For each condition ⊕ indicates increased
diﬃculty, ⊖ indicates decreased diﬃculty.
to show that this limitation can be overcome through the use of randomised questions generated on-the-ﬂy.

Vocabulary testing and drilling
Drills are a short, fast, throwaway form of testing in which the act of testing itself forms
part of the learning process. If testing is used to make decisions, then drills are used to make
immediate and short-term decisions about what to study next. Indeed, this description
matches very closely the ﬂashcard software we described earlier, in which this simple form of
testing is used. However, ﬂashcards limit drills to simple forms of questions, with ultimately
limited linguistic motivation. is section asks, what types of questions are used in testing,
and which are promising for automated testing?
A popular form of testing are vocabulary levels tests, as discussed earlier, which test
receptive knowledge of words (Nation 2001). In extension to this form of testing, Laufer
and Goldstein (2004) describes the CATSS (Computer Adaptive Test of vocabulary Sizeand Strength) test. Instead of our earlier view of vocabulary knowledge as multidimensional,
Laufer and Goldstein considers knowledge of a word to be unidimensional but continuous,
ranging from superﬁcial familiarity to the ability to use the word correctly in open speech.
Furthermore, she aligns the receptive/productive knowledge distinction along this axis ofword proﬁciency, and considers four scenarios for the test questions, as given in Table 3.4.
Let us use 山 yama “mountain” as our example, with English the L1 and Japanese theL2. In the active condition, the L1 meaning “mountain” is given, and the learner has to62

Chapter 3: Learning Japanese vocabulary

either recall the L2 form 山 or recognise it amongst distractors. In the passive condition
the L2 form 山 is given and the learner has to either supply its L1 meaning “mountain”
or recognise amongst distractors (e.g. “city”, “hill”, “vehicle”, “river”). ese correspond tothe hypotheses that accessing the meaning from the form is easier than accessing the form
from the meaning, and likewise that recognising a word is easier than producing it.
CATSS is adaptive in that, having hypothesised these levels of word knowledge, thelower levels of knowledge are only tested if a higher level has failed. Furthermore, the test
is structured into levels based on 1000-word frequency bands, which the learner is tested onserially. Although selection of an appropriate frequency band would seem a prime candidate
for further adaptive testing, instead CATSS allows individual learners to both skip ahead
to later levels, and to determine for themselves when they have reached their cut-oﬀ point
and stop the test.
In contrast to adaptive tests, we now consider drills, and their relationships to these tests.
Zock and Quint (2004) proposed developing drill tutors from dictionaries based on pattern
drills. Since Japanese has freely available dictionary resources, it seems a prime candidate
for such drills. is work was later continued (Zock and Afantenos 2007) and developed
into a system for drilling basic conversational patterns. ese patterns aim to not just aid
in vocabulary growth, but to improve automaticity of knowledge, an additional commonly
postulated dimension of vocabulary knowledge.
If we compare such drills to formalised tests, several factors emerge. Formalised tests
often have much at stake for learners, so the emphasis on the reliability and validity oftest outcomes is large. is means that strong tests are developed and evaluated for theirdiscriminating ability amongst candidates. In contrast, drills are typically characterised byﬂashcard-like simplicity, and are prized for their variety and importantly their coverage of
the words under study. Since drill systems are used as vocabulary study methods, their users
required this coverage to ensure that their desired vocabulary is acquired.
e amount of manual work required to generate the more linguistically interesting
test questions – for example those used the Simple Kanji Aptitude Test (SKAT) describedby Toyoda and Hashimoto (2002) – means that full coverage will never be achieved bythese traditional methods. Furthermore, the manual labour involved and subsequent limited number of test questions means that these tests are impractical for use as a means ofChapter 3: Learning Japanese vocabulary63

self-study; the number of tests available is small, and after the ﬁrst exposure, subsequent
exposures have greatly weakened validity. However, the ﬂashcard drills which allow timely
self-evaluation lack the sophistication of current adaptive tests which use statistical modelling to quickly hone in on learner aptitude.
One of the main contributions of this thesis is in bridging these two methods of testing,and generating linguistically motivated drills which have the potential for use in adaptive
testing. Questions for these drills are aimed to be equivalent to questions manually developed for paper tests or computer adaptive tests, but by generating these questions automatically, they also have the coverage and availability required to be used successfully in learner
drills.

3.5

Conclusion

In this chapter we discussed a range of issues relating to vocabulary study for Japanese.
In particular, we argued that supporting the early reading process is the best way to help
learners attain the large vocabulary they need for ﬂuency. Of the many lexical relationships available, we identiﬁed near-homography as a promising resource yet to be utilised
for aiding learner study. In Chapter 4 to follow, we examine near-homography in depth
and attempt to determine the accessibility of a word’s orthographic neighbourhood. We
also argued that dictionary resources and testing could both be greatly improved with better
error modelling. Graphemic relationship models from Chapter 4 then provide a basis forextending the FOKS dictionary to allow partial knowledge search by kanji in Chapter 5.
Ultimately, we combine both phonetic and graphemic confusion models into a system foradaptive testing in Chapter 6, before concluding with our overall ﬁndings.

Chapter 4
Orthographic similarity
4.1

Introduction

Overview
Chapter 3 has made clear the need for more accessible dictionaries for languages such as
Japanese and Chinese. We now concern ourselves with the perceptual process of identifying
characters, in particular the behaviour of perception within dense visual neighbourhoods.
Within the dictionary accessibility space, we are motivated by the potential to correct confusion errors, but also to leverage the mental associations provided by visual proximity toallow advanced learners to ﬁnd unknown characters faster. We develop this form of lookup
later in Chapter 5.
In order to do so, we require some formal notion of the distance between two charactersso as to distinguish near-neighbours, which are plausibly confusable or mentally associated,from higher distance pairs. is chapter then concerns itself with accurate modelling ofgraphemic similarity.

Distance within alphabetic orthographies
Within alphabetic orthographies such as English, distance metrics are used at the wordlevel or higher, rather than at the character level. is is natural, since alphabetic characters individually do not have associated semantics. At the word-level, the most common
64Chapter 4: Orthographic similarity

65

distance metric is Levenshtein distance (or edit distance). e most natural application isspelling correction, since words with spelling errors are usually visually similar to their correct form.

Sentence level

Word level

EnglishJapanese

The waiter misunderstood my order.

私は医者になった。

waitermisunderstood

は

my

医者

医
mis-

Morpheme level

Character levelmiSentence levelに者

Word level

Character level

understood

s

!

匚

矢

一

一

Component level!

"

Stroke level

Figure 4.1: Drawing rough equivalence between the linguistic units used in English andJapanese.
Figure 4.1 provides a comparison of the linguistic units in English and Japanese. In
Japanese, the individual kanji level is roughly equivalent to the morpheme level for English,
with free morphemes as 日 hi “day” which are words in their own right, and bound morphemes such as 向 which can only occur as part of a larger word. For example, 向 can occur
with conjugational suﬃxes (向かい mukai “facing” or 向かう mukau “to face”) or as partof larger compounds (向心力 kōshiNryoku “centripetal force”). Although the stroke level inJapanese roughly aligns to the character level in English, it lacks both the phonemic contribution to whole-word pronunciation and the linear spatial continuity of characters within
an English word.
Between strokes and characters lie common stroke groupings known as radicals. Some
radicals are themselves characters, such as 日 hi “day/sun” and 月 gatsu “month/moon” within
明 aka(rui) “bright”. Others are known short-form variants of full kanji, for example theradical 扌 as a short form for 手 te “hand” and 氵 as a short form for 水 mizu “water”.
Finally, some radicals have no whole-character equivalents, though their semantics are well

66

Chapter 4: Orthographic similarity

known, such as 艹 “grass”. So far all example radicals presented have been atomic. However,combinations of radicals themselves may be considered radicals with stable semantics orphonetics.
e complexity of kanji compared to English morphemes or words thus demands new
methods of measuring similarity.

4.2

Distance models: a ﬁrst approach

Form of models
We can formalise our distance models as follows. Let K be the set of all kanji. A
distance model is a function d : K × K → R+ , mapping kanji pairs to the value of the
distance between them. In practice, each d is composed of two functions ϕ : K → F andd′ : F × F → R+ , where ϕ maps each kanji to its representation in some intermediate
feature space F , and d′ is a distance function on F .
ere are three additional desirable constraints on our choice of d. We require, for all
ka , kb , kc ∈ K:d(ka , kb ) = 0 ⇐⇒ ka = kbd(ka , kb ) = d(kb , ka )d(ka , kc ) ≤ d(ka , kb ) + d(kb , kc )identity of indiscernibles(4.1)

symmetry(4.2)

triangle inequality(4.3)

Equation 4.1 ensures that d can distinguish between diﬀerent characters, Equation 4.2
meets our intuition that similarity is a symmetric relationship between characters, andEquation 4.3 simply imposes additional regularity on the geometry that d generates on
K. If these three requirements are met, then (K, d) is a metric space, and d is a metric on K.
ese requirements in turn constrain our intermediate functions ϕ and d′ . To satisfy
them, d′ should also be a metric on F . Furthermore, if two kanji have an identical feature
representation, then d′ will not be able to distinguish between them, and thus their composition will not meet Equation 4.1. For this reason, ϕ must be injective, mapping each kanji
to a unique representation in F .
Having scoped possible forms of d, we now consider two simple distance models.

Chapter 4: Orthographic similarity

67

Bag of radicals model
We have established that radicals are highly salient features of kanji, as components
with their own semantic and phonemic information. Indeed, they are the primary meansof looking up an unknown character using the traditional lookup method.1
At the radical level, the primary data set available is radkﬁle,2 which provides radicalmembership data for each kanji, and serves as the basis of the WWWJIC multi-radical
kanji lookup system.3 e natural feature space is thus that of sets of radicals, where kanji
are decomposed into their constituents, as shown in Figure 4.2.

新 → {八, 立, 十, 辛, 木, 斤}
薪 → {艹, 八, 立, 十, 辛, 木, 斤}
Figure 4.2: Kanji decomposed using radkﬁle into their naive radical-member feature set
representations.

If we take the natural choice of ϕ, mapping kanji directly to their radical sets, we immediately ﬁnd that two kanji can have the same feature representation. Examples of kanji
which are mutually indistinguishable in this manner are shown in Table 4.1. is occurs
for several reasons.
Firstly, radical membership naturally discards both position and number of radicals, andmany small sets of kanji can be found which diﬀer only in these aspects. For example, 木,
林 and 森 only diﬀer in the number of the same primary radical (木) used, and are thus
considered identical. Similarly, kanji pairs in which only the position of the radicals diﬀer,
for example with 略 and 畧 (radicals: 田, 口, 夂).
e ﬁrst known Chinese dictionary to index via radicals was the 说文解字 [shūowénjǐezì], a Chinese
dictionary written in the 1st century A.D. by 许慎 [xǔshèn]. It chose one radical from each character, usually
the left-most or top-most semantic radical, as the header under which the character could be found.
2
http://www.csse.monash.edu.au/~jwb/kradinf.html
3
http://www.csse.monash.edu.au/~jwb/cgi-bin/wwwjdic.cgi?1R1

68

Chapter 4: Orthographic similarity
Identity group
Radicals
木, 林, 森木田, 口, 夂略, 畧拐, 招扌, 口, 刀万, 丑, 乃, 乍, 垂 ノ, 一, ｜Reasonnumber of 木 radicals
position of 田 radical
position of 口 and 刀 radicals
ambiguity of subcomponents

Table 4.1: Groups of characters which share an identical feature-set representation according to radkﬁle.

Secondly, there is some small amount of noise generated by the ambiguity in the transition from the stroke level to the radical level. For example, 乃 nai “whereupon” is itself
a radical, but is listed as containing the single-stroke radicals ノ, 一, and ｜. Whilst these
may be useful from a dictionary lookup perspective, from a similarity perspective they add
noise. e vast majority of characters are either themselves simple non-stroke radicals orare built completely from such radicals, but there are still many characters for which sucha description is inaccurate, since they either contain known radicals with extra strokes, orthey are composed of stroke groups too rare to be considered radicals.
We get around both problems by adding each kanji to its radical-set, using
ϕradical (k) = radicals(k) ∪ {k}(4.4)

is ensures a unique representation for each kanji. We can now add a distance metriconto our feature space. We choose a function based on cosine similarity, a commonly usedmeasure in information retrieval, as given in Equation 4.5 below.
dradical (x, y) = 1 −|ϕradical (x) ∩ ϕradical (y)||ϕradical (x)||ϕradical (y)|(4.5)

Having focused on modelling at the radical level, we now consider the whole-kanji level asan alternative means of modelling similarity.

L1 norm across rendered images
In Section 3.2, we brieﬂy discussed evidence for stroke level processing in visual character recognition. Indeed, the limited real-life confusability data which is available suggeststhat stroke-level contributions to whole-character similarity are a source of confusion. For

Chapter 4: Orthographic similarity

69

example, a native speaker analysis of FOKS error logs determined that 基 ki, moto “basis”
and 墓 bo, haka “grave/tomb” were confused by a learner during search (Bilac et al. 2003).
is example shows that learners can mistake very similar looking kanji with few if any
shared radicals, provided there are suﬃcient similar looking strokes in similar positions.
e ultimate representation of how kanji are displayed to users are the actual pixels which
will are displayed on-screen. ese will vary by font, colour, size, placement and context.
We naturally abstract away context, and ignore size and colour by rendering each kanji toa ﬁxed-size black and white image. By choosing rendered images (or rather the intensity
for each pixel in that rendered image) as our feature space, we are discarding any additional
knowledge we may have about each kanji’s internal structure, and instead considering themas arbitrary symbols.
Image distance metrics may be used directly for image similarity search (Zhang and Lu
2003), or embedded within image recognition algorithms such as Radial Basis Function
Support Vector Machines, Principal Component Analysis and Bayesian Similarity (Wanget al. 2005). For many applications, distance models must typically be scale-, translationand rotation-invariant (Yang and Wang 2001), transformations which humans easily ignore
but which require sophisticated algorithms to counteract. Fortunately, these issues do notoccur in our limited feature space: Chinese characters occupy regular sized blocks whentypeset, whereas roman characters are variable sized, and must undergo kerning duringtypesetting to ensure the perceived spacing is even between characters. is diﬀerence also
exists at a pedagogical level: practice books for children learning to write English provide
horizontal lines so that they learn to write with uniform height, whereas Japanese practice
books contain a square grid so that children learn to write with uniform size.
Since kanji are rendered aligned with each other, it suﬃces to use a simple image distancemetric. We thus choose the L1 norm from the family of Lp norms known as Minkowski
distances. e L1 norm in particular is also known as the Manhattan or taxi-cab distance,
and is a simple baseline in image similarity literature:

L1 (x, y) =∑|px (i, j) − py (i, j)|

(4.6)

i,j

e L1 norm is very simple to calculate, but is known to be sensitive to relatively small

70

Chapter 4: Orthographic similarity

image perturbations. For our purposes, its output may be aﬀected by changes to the rendering method, in particular the size of the rendered images and the font used for rendering.
We considered an image size of 80×80 pixels to be suﬃciently detailed, and used this in all
experiments described here. To attempt to attain reasonable font independence, the same
calculation was done over 5 commonly available fonts, then averaged. e fonts used were:
Kochi Gothic (medium gothic), Kochi Mincho (thin mincho), Mikachan (handwriting),
MS Gothic (thick gothic), and MS Mincho (thin mincho). e graphics program Inkscape4
was used to render them non-interactively.
We expect that the L1 norm is likely to underestimate the perceptual salience that repeated stroke units (i.e. radicals) have, and thus underestimate radical-level similarity, except where identical radicals are well aligned. Nevertheless, we expect it to correlate wellwith human responses where stroke-level similarity is present. Pairs scored as highly similarby this method should thus also be rated as highly similar by human judgements.
We now discuss an experiment aimed at collecting data to evaluate these two metrics.

4.3

Similarity experiment

Experiment outline
In order to eﬀectively evaluate our preliminary similarity models, we conducted an exploratory web experiment with the aim of collecting a set of gold-standard orthographic
similarity judgements. Participants were ﬁrst asked to state their ﬁrst-language background,and level of kanji knowledge, pegged to one of the levels of either the Japanese Kanji Aptitude Test5 or the Japanese Language Proﬁciency Test.6 Participants were then exposed toa number of pairs of kanji, in a manner shown in Figure 4.3, and asked to rate each pair
on a ﬁve point graded similarity scale. e number of similarity grades chosen represents a
trade-oﬀ between rater agreement, which is highest with only two grades, and discrimina4

http://www.inkscape.org
日本漢字能力検定試験: e Japanese government test of kanji proﬁciency intended for native speakers,
which is initially tied to Japanese grade school levels, but culminates at a level well above that expected in highschool graduates.
6
日本語能力試験: e Japanese government general-purpose test of Japanese aptitude for secondlanguage learners of Japanese. e test is administered by the Japan Foundation.
5

Chapter 4: Orthographic similarity

71

tion, which is highest with a large number of grades. Each kanji displayed was rendered as
a 50×50 pixel image in MS Gothic font, for consistency across browsers. Whilst this did
not guarantee a ﬁxed visual size for each kanji rendered, since participants may have used
computer displays of diﬀering spatial pixel density, it was nonetheless expected to reducethe variation in a kanji’s visual size across participants.
Although participants included both ﬁrst and second language readers of Chinese, only
Japanese kanji were included in the stimulus. Chinese hanzi and Japanese hiragana andkatakana were not used for stimulus, in order to avoid potential confounding eﬀects ofcharacter variants and of diﬀering scripts. e pairs were also shuﬄed for each participant,with the ordering of kanji within a pair also random, in order to reduce any eﬀects causedby participants shifting their judgements part-way through the experiment.

Figure 4.3: Example stimulus pair for the similarity experiment. is pair contains a sharedradical on the left.

Each participant was exposed to a common set of 65 control pairs, to be discussed in
Section 4.3 below. Further, a remaining 100 random kanji pairs were shown where bothkanji were within the user’s speciﬁed level of kanji knowledge (where possible), and 100
were shown where one or both kanji were outside the user’s level of knowledge. is wasin order to determine any eﬀects caused by knowing a kanji’s meaning, its frequency, its
readings, or any other potentially confounding properties.
Web-based experiments are known to provide access to large numbers of participants
and a high degree of voluntariness, at the cost of self-selection (Reips 2002). Althoughparticipants of all language backgrounds and all levels of kanji knowledge were solicited,
the nature of the experiment and the lists advertised to biased participants to be mainly of
an English, Chinese or Japanese ﬁrst-language background.

72

Chapter 4: Orthographic similarity

Control pairsere are many possible inﬂuences on orthographic similarity judgements which we
hoped to detect in order to determine whether the data could be taken at face value. A
sample pair and a description of each control eﬀect is given in Figure 4.4. Since the numberof potential eﬀects considered was quite large, the aim was not statistical signiﬁcance forthe presence or absence of any eﬀect, but rather guidance in similarity modelling should
any individual eﬀect seem strong. All frequency and co-occurrence counts were taken from1990–1999 Nikkei Shimbun corpus data.

Results
e experiment had 236 participants, with a dropout rate of 24%. e participants
who did not complete the experiment, and those who gave no positive responses, wereﬁltered from the data set. e remaining 179 participants are spread across 20 diﬀerent
ﬁrst languages. Mapping the responses from “Very diﬀerent” as 0 to “Very similar” as 4,
the mean response over the whole data set was 1.06, with an average standard deviation foreach stimulus across raters of 0.98. e full data set is available in unﬁltered form online.7
To measure the inter-rater agreement, we consider the mean rank-correlation across all
pairs of raters. Although the kappa statistic is often used (Eugenio and Glass 2004), it
underestimates agreement over data with graded responses. e mean rank correlation for
all participants over the control set was strong at 0.60. However, it is still lower than thatfor many tasks, suggesting that many raters lack strong intuitions about what makes one
kanji similar to another.
Since many of the ﬁrst language backgrounds had too few raters to do signiﬁcant analysis on, they were reduced to larger groupings of backgrounds, with the assumption that
all alphabetic backgrounds were equivalent. Firstly, we group ﬁrst-language speakers ofChinese (CFL) and Japanese ( JFL). Secondly, we divide the remaining participants fromalphabetic backgrounds into second language learners of Japanese ( JSL), second language
learners of Chinese (CSL), and the remainder (non-CJK). Participants who studied bothlanguages were put into their dominant language based on their comments, or into the JSL
7

http://ww2.cs.mu.oz.au/~lljy/datasets/#kanjiexp

Chapter 4: Orthographic similarity

73

Eﬀect type
Frequency (independent)

Example会店

Co-occurrence

法考

Homophones

弘博

Stroke overlap

策英Shared graphemes

働動Shared structure

幣哲Stroke count

奮撃Part of speech/function

方事

Semantic similarity

千万

Description
Frequency of occurrence of each kanji individually. Both kanji in the example pairare high-frequency.
Both kanji co-occur with high frequency
with some third kanji. For example,
法 hō “Act (law)” occurs in 法案 hōaN “bill(law)”, and 考 kaNga(e) “thought” occursin 考案 kōaN “plan, idea”.
Both kanji share a reading. In the example, both 弘 hiro(i) “spacious” and博 haku “doctor” share a reading hiro. For
博 this is a name reading.
Both kanji share many similar strokes, although no radicals are shared.
Both kanji share one or more graphical elements. ese elements might occur inany position.
Both kanji share the same structural
break-down into sub-components, although the sub-components diﬀer.
Pairs comparing and contrasting stroke
counts. Both examples here have a very
high stroke count.
Both kanji have a common syntactic function.
Both kanji are semantically similar. In theexample, they are both numbering units.

Figure 4.4: Groups of control pairs used, with an example for each. Parts of readings inbrackets indicate okurigana, necessary suﬃxes before the given kanji forms a word.

179
88
57
18
10
6

74

mu-sig-stim
Mean response mu-sig-rater
1.06
1.04
1.44
1.15
0.72
0.97
0.57
0.86
0.8
0.96
0.55
0.74

c-mean
0.98
0.99
0.73
0.78
0.68
0.99

Mean pairwise rank correlation

Mean response

0.75
0.50
0.25
0

All

Non-CJK

JSL

JFL

CSL

CFL

rc-r-std
Mean rank correlation
0.6
0.59
0.65
0.69
0.64
0.64

rc-p-mean
0.01
0
0
0.01
0
0

0.80

0.20

0.70

0.15

0.60

0.07
0.02
0
0.04
0
0.01

0.10

0.05
0.50
0
0.40

All

Non-CJK

Language group

JSL

JFL

CSL

CFL

Language group

Figure 4.5: Participant responses grouped by language background, measured over the control set stimulus. On the left we give the mean response for each group; on the right, themean pairwise rank correlation between raters of the same group.

group in borderline cases.8
Figure 4.5 shows mean responses and agreement data within these participant groups.
is grouping of raters is validated by the marked diﬀerence in mean responses across thesegroups. e non-CJK group shows high mean responses, which are then halved for second
language learners, and lower still for ﬁrst language speakers. Agreement is higher for theﬁrst-language groups ( JFL and CFL) than the second-language groups ( JSL and CSL),
which in turn have higher agreement than the non-speakers. Both of these results together
suggest that with increasing experience, participants were more discerning about what theyfound to be similar, and more consistent in their judgements.

Evaluating similarity models
Normally, with high levels of agreement, we would distil a gold standard data-set ofsimilarity judgements, and evaluate any model of kanji similarity against our gold-standard
judgements. Since agreement for the experiment was not suﬃciently high, we instead evaluate a given model against all rater responses in a given rater group, measuring the mean
rank-correlation between the model and all individual raters in that group.
8

rc-p-std

0.13
0.11
0.1
0.17
0.07
0.22

Inter-rater agreement within language groups

Mean response across language groups

1.00

0.16
0.13
0.13
0.19
0.08
0.2

Chapter 4: Orthographic similarity

1.50
1.25

c-std
0.53
0.55
0.6
0.57
0.64
0.16

Kappa

N
All
Non-CJK
JSL
JFL
CSL
CFL

Many alternative groupings were considered. Here we restrict ourselves to the most interesting one.

All

Non-CJK

Chapter 4: Orthographic similarity

75

We also have reference points to determine good levels of agreement, by measuring theperformance of the mean rating and the median rater response this way. e mean rating fora stimulus pair is simply the average response across all raters to that pair. e median rater
response is the response of the best performing rater within each stimulus set (i.e. the most“agreeable” rater for each ability level), calculated using the above measure.

Model evaluation
e pixel and radical models were evaluated against human judgements in various participant groups, as shown in Figure 4.6, and can be compared to the mean rating and medianraters. e pixel based similarity method exhibits weak rank correlation across the board,but higher correlation with increasing kanji knowledge. e radical model however shows
strong rank correlation for all groups but the non-CJK, and better improvements in theother groups.
ese results match our predictions for the pixel-based approach, in that it performs
reasonably, but remains only an approximation. e radical method’s results, however, areof a comparable level of agreement within the CFL and JFL groups to the median rater,a very strong result. It suggests that native speakers, when asked to assess the similarityof two characters, make their judgements primarily based either on the radicals which areshared between the two characters. Intuitively, this makes sense. Native speakers have thegreatest knowledge of radicals, their meaning, and their semantic and phonetic reliability.
ey also have the most experience in decomposing kanji into radicals for learning, writing
and dictionary lookup.
Group
Mean Median
L1
Non-CJK
0.69
0.55 0.34
CSL
0.60
0.65 0.38
CFL
0.51
0.62 0.44
JSL
0.64
0.70 0.43
JFL
0.56
0.69 0.46
All
0.65
0.62 0.39

dradical
0.47
0.56
0.66
0.59
0.68
0.54

Figure 4.6: Rank correlation of pixel and radical models against raters in given participantgroups. Mean and median raters provided as reference scores.

76

Chapter 4: Orthographic similarity

Band
Mean Median[ 0,1) 0.69
0.55[ 1, 200) 0.62
0.60
[ 200, 600) 0.64
0.69
[ 600, 1000) 0.69
0.72[1000, 2000) 0.56
0.70[2000, ...) 0.58
0.73

L1
0.34
0.38
0.41
0.46
0.46
0.48

dradical
0.47
0.53
0.61
0.52
0.65
0.70

Figure 4.7: Rank correlation of pixel and radical models against raters in across bands ofkanji knowledge. Each band contains raters whose number of known kanji falls within thatband’s range.

1000
mean

900

pixel
bag of radicals

800

Frequency

700
600
500
400
300
200
100
0
0

0.1
0.05

0.2
0.15

0.3
0.25

0.4
0.35

0.5
0.45

0.6
0.55

0.7
0.65

0.8
0.75

0.9
0.85

1
0.95

Similarity

Figure 4.8: Histograms of scaled responses across all experimental stimulus pairs, takenfrom mean rating, pixel and bag of radical models. Responses were scaled into the range[0, 1].

Chapter 4: Orthographic similarity77

e radical model still has poor correlation with the non-CJK group, but this is not an
issue for applications, since similarity applications primarily target either native speakers or
learners, who either already have or will pick up the skill of decomposing characters into
radicals. To attempt to determine when such a skill gets picked up, Figure 4.7 shows agreement when raters are instead grouped by the number of kanji they claimed to know, basedon their proﬁciency level. Aside from the [600, 1000) band, there are consistent increases
in agreement with the radical method as more kanji are learned, suggesting that the change
is gradual, rather than sudden. Indeed, learners may start by focusing on strokes, only toshift towards using radicals more as their knowledge of radicals improves.
e response histograms in Figure 4.8 show stark diﬀerences between human responses
and the two models. e radical model considers the majority of stimuli to be completely
dissimilar. Once it reaches stimulus pairs with at least one shared radical, its responses arehighly quantised. e pixel model in comparison always ﬁnds some similarities and some
diﬀerences, and is distributed normally. Human responses for our experiment lie somewhere in between the pixel and radical models, featuring a much smaller number of stimuli
which are completely dissimilar, and a shorter tail of high similarity than found with the
pixel model. A potentially signiﬁcant parameter which we do not investigate here is therelationship between the human response histogram and the visual size of kanji stimulus
presented. One participant commented that they commonly confuse kanji for one another,but only at small font sizes, not at the size presented in our experiment. It remains open
whether the same experiment with kanji rendered to smaller sizes would have elicited increased judgements of high similarity, and equally the extent to which noise (as measured
by rater agreement) is aﬀected. Full consideration of this parameter is beyond the scope of
our investigation.

4.4

Distance models: a second approach

Our ﬁrst attempts at modelling similarity gave several insights, relating to both modelling and to evaluation. Our radical metric captured coarse-grained salient features of kanji,
but largely ignored the organisation of these features within a kanji. Our pixel model in contrast responded well to structure, but more poorly to salient substructure. In this section we

78

Chapter 4: Orthographic similarity

describe an improvement to the radical model and two new metrics which aim to reach a
middle ground between these features. Our previous experiment sampled the space of kanji
pairings randomly for the majority of human judgements sought. is was done in order toremove potential bias from human selection of pairs. However, what this unintentionallyhighlighted was the rarity of high similarity pairs: the vast majority of characters are simply
reasonably distinctive from one another.
For any given character, there appear to be only a handful or fewer high-similarity neighbours for which it might be reasonably confused. We are thus virtually assured that a randomly chosen pair of kanji will be distinctive from one another, rather than similar. Our
random sampling thus had the unintended eﬀect of limiting our measurement to low similarity pairs. For applications, detection and accuracy over high similarity pairs is far more
important. For this reason, we introduce two new data-sets for evaluation, and compare
metrics over these data sets.

ModelsBag of radicals with shape
We discussed earlier the salience of radicals, and developed our bag-of-radicals model
dradical . However, this model ignores the position of radicals, which is known to be importantin similarity judgements, and also the number of each radical within a kanji. To address
multiplicity of radicals and the ﬁndings of Yeh and Li’s (2002) study, we set the above metricto unit distance whenever the two characters diﬀer in their basic shape. We approximate
the broad shape by the use of the ﬁrst part of each kanji’s 3-part SKIP code, which cantake values horizontal, vertical, containment or other. SKIP codes for each kanji are provided
by kanjidic (introduced in Section 3.1), and radical membership by radkﬁle (introduced in
Section 4.2).
is change allows the adjusted metric, dradical+shape , to distinguish between examples
with repeated components. e ﬁnal metric aims to capture the visual and semantic salienceof radicals in kanji perception, and to also take into account some basic shape similarity.

Chapter 4: Orthographic similarity

dradical79

持扌, 土, 寸待彳, 土, 寸

乗

L1

東
日

丨, ㇆, 一, 一目

丨, ㇆, 一, 一, 一

持

待

持

待

扌 寺

彳 寺

dstroke

dtree
2

8

6

土 寸

2 3a

2

2

4 4 3
8

1

土 寸

2 3a

2

2

Figure 4.9: A summary of our kanji distance metrics

8

1

80

Chapter 4: Orthographic similarity

Stroke edit distance
We saw earlier in Figure 4.1 that just as individual letters occupy the lowest orthographic
level for English, individual strokes do the same in Japanese. Spelling correction in English
has largely focused on mistakes at the letter level, since computer input for English is mediated by the keyboard, and the keyboard is a per-letter input device. In Japanese, withcomputer input mediated by IME software, stroke errors in input can not occur unless the
mistaken character is also a homophone of the desired character. Without such a sourceof errors to correct, stroke-based distance metrics have not to our knowledge been examined for Japanese. However, since strokes are the lowest unit from which radicals and kanji
are constructed, metrics focusing on the stroke level should be able to capture very ﬁne
distinctions between characters, and thus distinguish between highly similar pairs.
A major obstacle to such metrics historically has been the lack of data sets for kanji
which describe them at the stroke-level. Fortunately, a hierarchical data set for Japanese
kanji was created by Apel and Quint (2004) which provides such a description. Each kanji
is speciﬁed by its strokes, grouped into common stroke groups (components), and broken
down in a hierarchical manner into relative positions within the kanji (for example: left
and right, top and bottom). e strokes themselves are based on a taxonomy of some 26
stroke types (46 including sub-variants). Each kanji has a ﬁxed stroke order, a single correct
order in which the strokes are hand-written to construct the kanji correctly. e strokes areprovided by the dataset in this order.
Ideally, we might wish to transfer successful distance metrics for alphabetic orthographies and examine their performance in Japanese. e most common metric used is edit
distance, which is the minimum cost over all edit paths – sequences of insertions, deletions
and substitutions which transform one string into another. How might we adequately represent kanji as linear strings, given their two-dimensional layout and the hierarchical data
available?
Although kanji are two-dimensional, the order in which strokes are written is ﬁxed not
only for each character, but also for each component reused across characters. e ordered
list of strokes used to write a character thus provides the representation we need to use edit
distance, and eﬀectively serves as a signature for each character. In terms of Apel and Quint’sChapter 4: Orthographic similarity

81

data set, this signature is formed from the leaves of each kanji’s tree representation, which
represent strokes. Taking the edit distance over such signatures gives our dstroke metric.
Figure 4.9 shows example signatures for 日 hi “sun” and 目 me “eye”, where each stroke
type is represented by an alphanumeric code. For example, the signature for 日 is the ordered sequence of strokes {丨, ㇆, 一, 一}, internally represented by the alphanumeric codes{3, 11a, 2a, 2a}. In its generalised form the edit distance is the minimum cost over all edit
paths from one string to another, and the cost of edit operations may be arbitrary andpositive. However, we make the simplifying assumption that all strokes types are equally
distinctive from one another. is allows us to use a unit cost for all edit operations. We
also normalise the edit distance so as to avoid penalising larger kanji due to their large stroke
count alone.
Although we discard structural features of each character to create its signature, much
useful information remains preserved. Since radicals within each character are written insequence, they form contiguous blocks within stroke signatures. e edit distance thus
recognises shared radicals whenever their position is similar enough. e order of radicals blocks in a signature also reﬂects their position as part of the larger compound, since
components are usually drawn in a left-to-right, top-to-bottom order. Finally, it provides asmooth blending from stroke similarity to radical similarity, and can recognise the similarity
between pairs like 日 hi “sun” and 目 me “eye”.

Tree edit distance
We discarded much of the hierarchical data to generate stroke signatures, in order touse string edit distance as a metric. is carries the implicit assumption that hierarchicalinformation is not as useful as stroke information when measuring similarity. To test thisassumption, we also created an additional metric which used the entire tree representation:
the ordered tree edit distance between kanji tree representations, dtree .
Tree edit distance is deﬁned as the minimum cost edit path between one tree and another, where an edit path is any sequence of node insertions, deletions and relabellings whichtransforms one tree into another (Bille 2005). ese operations mirrors closely the edit
operations used in string edit distance. As with string edit distance, a cost function can be

82

Chapter 4: Orthographic similarity

speciﬁed giving arbitrary positive symbol-dependent costs to each type of edit operation.
We again make a simplifying assumption, that all structural aspects are equally important,allowing us to use unit cost for each edit operation. Figure 4.9 provides an overview of thestructure of each kanji’s tree representation, though actual trees we generate also containphonetic elements, radicals, and stroke groups whose strokes are in non-contiguous blocks.
For our kanji trees, generalised tree edit distance completely subsumes stroke edit distance. By this we mean that tree edit distance can perfectly emulate stroke edit distanceif an appropriate cost function is chosen, say one which gives zero-cost edit operations fornon-stroke nodes. is is possible regardless of the cost function used for stroke edit distance. From this perspective, a comparison between these two methods is only a comparison
between cost functions for tree edit distance.
However, both the algorithmic complexity and the implementation of string edit distance are far simpler than that of tree edit distance. For two kanji i and j, let si be thenumber of strokes in kanji i, and ni is the number of nodes in its tree representation. en
stroke edit distance takes O(si sj ) time. In comparison, using Demaine et al.’s (2007) optimal decomposition algorithm – which the authors generously provided sample code for –our tree edit distance implementation had worst-case time complexity O(ni n2j (1+log( nnji ))).
is is a signiﬁcant improvement over earlier algorithms, such as those described by Bille(2005), but in practice our implementation remained two orders of magnitude slower thanthat for string edit distance.

Evaluation
We evaluated all four kanji distance metrics over three data sets. Firstly, we re-evaluatedover the earlier similarity experiment data (from Section 4.3) in order to compare the newly
added metrics against those already discussed. However, this data-set had the acknowledged
weakness of not including enough high-similarity pairs. We thus sought to rectify this byﬁnding a source of expert similarity judgements, ﬁnding these judgements in the form of
the White Rabbit JLPT Level 3 kanji ﬂashcard set. Each ﬂashcard contains either one or
two highly-similar neighbours which might be confused with a given kanji. We use thisset to determine our likely performance in a search task. Finally, we elicited further nativeChapter 4: Orthographic similarity

83

speaker judgements in a similarity pool experiment, and again compared each of the metrics
on this new data set. We discuss each of these evaluations below.

Similarity experiment data
Calculating the rank correlation ρ averaged over raters in each group, as we did earlier,gives the results shown in Figure 4.10.

Metric agreement within rater groups
0.700

0.525

0.350

0.175

e

dr

ad
i

ca
l

dr

+s
h

ad
i

ap

ca

l

L1

e
tro
k

ds

dt

re
e

0

Non-CJK

JSL

JFL

Figure 4.10: Mean value of Spearman’s rank correlation ρ, calculated over each rater groupfor each metric.

e results show the same broad patterns as before in terms of language ability: the meanrank correlation increased as the participants’ knowledge of Japanese increased. However,the dradical+shape metric dominates over the other metrics, including the original dradical , at alllevels of knowledge. is conﬁrms the salience of radicals and the tendency for individuals
to classify kanji by their broad shape, as suggested by Yeh and Li (2002). L1 , dstroke anddtree perform poorly in comparison. Interestingly, despite large diﬀerences between themfor non-speakers, all perform equivalently for native-speakers.

84

Chapter 4: Orthographic similarity
Despite overall poor performance from our new metrics, we have been able to improveon dradical by adding shape information. We now describe the ﬂashcard data set, and evaluate
over it for comparison.
Flashcard data set
Having identiﬁed problems with our earlier experiment due to the lack of high-similarity
stimulus pairs, we looked for a source of such pairs. A series of kanji ﬂashcards developedby White Rabbit Press provided this source. Each card is designed to allow the study of asingle kanji, for example 晩 in Figure 4.11. For each kanji one or two visual neighbours areprovided whose visual similarity to the kanji being studied makes them potentially confusable. We used their JLPT 3 ﬂashcards, providing such visual neighbours for 245 kanji intotal. We have provided these similarity pairs online for further investigation and use.9

Figure 4.11: An example White Rabbit Press ﬂashcard for the kanji 晩 baN “evening”. Notethe two large characters on the right provided due to visual similarity with the main kanji.
We took two diﬀerent approaches to evaluation. Firstly, for each high-similarity pair(a pivot kanji and its distractor), we randomly selected a third kanji from the jōyō character set10 which we combined with the pivot to form a second pair. By virtue of the large
number of potential kanji pairings, most of which bear no real similarity, this second pair is
910

http://ww2.cs.mu.oz.au/~lljy/datasets/#whiterabbit
e “common use” government kanji set, containing 1945 characters.

Chapter 4: Orthographic similarity
MetricL1
dtree
dstroke
dradical
dradical+shaperandom baseline85
Accuracy
0.954
0.952
0.926
0.736
0.603
0.500

Table 4.2: Accuracy at detecting which of two pairs (ﬂashcard vs. random) has highsimilarity

highly likely to be of lower similarity than the ﬁrst pair. We then compared how well each
metric can classify the two pairs by imposing the correct ordering on them, in the form ofclassiﬁcation accuracy. e results of this form of evaluation are shown in Table 4.2. We
include a theoretical random baseline of 0.500, since any decision has a 50% chance of being
successful.
One immediate and surprising result is that the dradical+shape and dradical metrics which
dominated the previous task perform far poorer than the other metrics. is suggests thatthey are poor at distinguishing between high and medium similarity pairs, though our earlier
evaluation suggests they broadly order examples correctly across the whole spectrum, as
shown by its performance on the similarity experiment data. eir precision is simply too
low for these high-similarity cases, but it is precisely these cases we are interested in for
useful search and error correction.
e three other metrics have accuracy above 0.9 on this task, indicating the ease withwhich they can distinguish such pairs. However, this does not guarantee that the neighbourhoods they generate will be free from noise, since the real-world prevalence of highly
similar characters is likely to be very low. For applications, it is most important that the few
true high-similarity neighbours are accurately reported by the chosen metric, and are not
swamped by erroneous candidates.
To better evaluate the retrieval performance of each metric, we considered the main
ﬂashcard kanji to be a query and its neighbours ranked in order of proximity as the retrieved
documents for this query. We used the high-similarity ﬂashcard neighbours as the fullset of relevant documents, implicitly considering all other kanji as irrelevant documents.

86

Chapter 4: Orthographic similarity
Metricdtree
dstroke
L1
dradical+shapedradical

MAP
0.320
0.318
0.271
0.211
0.177

p@1
0.313
0.310
0.257
0.197
0.144

p@5 p@100.149 0.094
0.151 0.099
0.139 0.089
0.087 0.063
0.085 0.065

Table 4.3: e mean average precision (MAP), and mean precision at N ∈ {1, 5, 10} over
the ﬂashcard data

dtree
dstroke
L1
dradical
dradical+shapedtree
dstroke
L1 dradical
0.934
0.026 <0.001
0.934
0.027 <0.001
0.026 0.027- <0.001<0.001 <0.001 <0.001<0.001 <0.001 <0.001 0.254

dradical+shape<0.001<0.001<0.001
0.254-

Table 4.4: Pairwise signiﬁcance ﬁgures for MAP scores over the ﬂashcard data, shown as pvalues from a two-tailed Student’s t-test
For each query we calculated the average precision (AP) for the given metric, according toEquation 4.7, where P (r) is the precision at a rank cutoﬀ r and rel(r) is a binary function
set to 1 when the retrieved result at rank r is a relevant document.

AP =1nrelevant∑nretrieved

P (r) × rel(r)

(4.7)

r=1

Using all queries gives a vector of AP values for each metric. Taking the mean of each
vector yields the mean average precision (MAP) statistic for each metric. MAP is widelyused in Information Retrieval to evaluate document ranking methods. is makes it suitable for use with similarity metrics, since for many applications the most important featureof a metric is the ranking of neighbours which it provides. A strength of MAP is thus thatit evaluates metrics in a manner that corresponds closely to their intended use. A disadvantage is that it assumes that all relevant documents are equally relevant, whereas in practice
we may ﬁnd important diﬀerences in proximity even between high-similarity neighbours.
In practice this is not a limitation, since the high-similarity neighbours provided by theﬂashcard set are provided to us unranked.

Chapter 4: Orthographic similarity

87

Table 4.3 shows the MAP score for each metric, along with the precision at N neighbours, for varied N . e precision statistics conﬁrm the rough ranking of metrics foundin the earlier classiﬁcation task, with the radical-based metrics performing worst. In thismore diﬃcult task, the L1 norm is outperformed by dstroke and dtree . Pairwise signiﬁcanceﬁgures as given in Table 4.4 were calculated by performing a two-tailed Student’s t-test on
the average precision vectors for each pairing of metrics. Most diﬀerences in MAP scores
are signiﬁcant at the 95% conﬁdence level, with a few key exceptions that are telling of therelationship between each of the metrics. dstroke and dtree give a p value of 0.934, indicating
their diﬀerence is nearly identical for this task. e two radical-based methods give a pvalue of 0.254, which gives low conﬁdence that their performance genuinely diﬀers. Bothof these results are unsurprising, given the similarity between the two forms of edit distanceand the two radical-based methods.

Distractor pool experiment
e ﬂashcard data, though providing good examples of high-similarity pairs, suﬀers
from several problems. Firstly, the constraints of the ﬂashcard format limit the numberof high-similarity neighbours which are presented on each ﬂashcard to at most two; insome cases we might expect more. Secondly, the methodology for selecting high-similarity
neighbours appears subjective.
For these reasons, we conducted an experiment to attempt to replicate the ﬂashcard data.
100 kanji were randomly chosen from JLPT 3 set (hereafter pivots). For each pivot kanji,
we generated a pool of possible high-similarity neighbours in the following way. Firstly,
the pool was seeded with the neighbours from the ﬂashcard data set. We then added thehighest similarity neighbour as given by each of our similarity metrics. Since these could
overlap, we iteratively continued adding an additional neighbour from all of our metrics
until our pool contained at least four neighbours.
Native or native-like speakers of Japanese were solicited as participants. After performing a dry run, each participant was presented with a series of pivots and their pooled neighbours, as shown in Figure 4.12. eir task was to select the neighbours (if any) which might
be confused for the pivot kanji, based on their graphical similarity. e order of stimuli was

88

Chapter 4: Orthographic similarity

randomised for each rater, as was the order of neighbours for each pivot. Kanji were provided as 30×30 pixel images using MS Gothic font for consistency across browsers, andwith our earlier similarity experiment (Section 4.3).

Figure 4.12: Example stimulus from the distractor pool experiment. For each kanji on theleft-hand side, participants could mark one or more of the kanji on the right as potentiallyconfusable visual neighbours.

3 participants completed the experiment, selecting 1.32 neighbours per pivot on average,
less than 1.86 per pivot provided by the ﬂashcard data. Inter-rater agreement was quite
low, with a mean κ of 0.34 across rater pairings, suggesting that participants found thetask diﬃcult. is is unsurprising, since as native speakers the participants are experts at
discriminating between characters, and are unlikely to make the same mistakes as learners.
Comparing their judgements to the ﬂashcard data set yields a mean κ of 0.37. e full
dataset is available for scrutiny and further use online.11
Ideally, this data generates a frequency distribution over potential neighbours based on
the number of times they were rated as similar. However, since the number of participants
is small, we simply pooled the neighbours with high-similarity judgements for each pivot,
yielding an average of 2.45 neighbours per pivot. Re-evaluating our metrics on this data
gives the ﬁgures in Table 4.5.
Compared with the ﬂashcard data set, the ordering and relative performance of metrics
is similar, with dstroke marginally improving on dtree , but both outperforming L1 and dradical .
e near-doubling of high similarity neighbours from 1.32 to 2.45 is reﬂected by a corresponding increase in MAP and precision@N scores, though the eﬀect is somewhat reduced
as N increases.
11

http://ww2.cs.mu.oz.au/~lljy/datasets/#poolexp

Chapter 4: Orthographic similarity
Metricdstrokedtree
L1
dradical
dradical+shape

89
MAP
0.406
0.383
0.349
0.288
0.211

p@1
0.530
0.560
0.530
0.350
0.270

p@5 p@100.240 0.146
0.234 0.142
0.210 0.123
0.168 0.122
0.122 0.096

Table 4.5: e mean average precision (MAP), and mean precision at N ∈ {1, 5, 10} overthe pooled distractor data

dstrokedtree
L1
dradical
dradical+shapedstrokedtree
L1 dradical- 0.467
0.103 0.002
0.467- 0.368 0.023
0.103 0.368- 0.115
0.002 0.023 0.115<0.001 <0.001 <0.001 0.004

dradical+shape<0.001<0.001<0.001
0.004-

Table 4.6: Pairwise signiﬁcance ﬁgures for MAP scores over the pooled distractor data,
shown as p values from a two-tailed Student’s t-test
Upon checking pairwise signiﬁcance tests for MAP calculations over this new data set(Table 4.6), some diﬀerences become clear. e performance improvement between dradical
and dradical+shape is signiﬁcant at the 95% conﬁdence level, rather than the weaker conﬁdence
shown over the same measurement for the ﬂashcard data. e diﬀerences between dstroke ,dtree and L1 are also reduced in signiﬁcance. Together, these changes could indicate a diﬀerence between the ﬂashcard and pooled distractor data, perhaps reﬂecting the diﬀerent biases
provided by the experimental participants in comparison to the creators of the ﬂashcard set.

4.5

Discussion

Modelling of graphemic similarity was initially hampered by a lack of data. In this chapter we have identiﬁed the White Rabbit Press ﬂashcards as a source of expert judgements,
and developed data sets for evaluating similarity metrics through two diﬀerent experiments.
is area is no longer data poor, and we anticipate the development of better metrics stillbased on this data.

90

Chapter 4: Orthographic similarity
Stroke edit distance and tree edit distance performed best on nearly all our evaluation

methods on high-similarity pairs, and indeed were roughly comparable for each task. issuggests that the stroke signatures alone capture much of the structural information important to whole-character similarity, or alternatively that both metrics were able to provide anappropriate level of fuzzy matching between similar components. Fortunately, both metrics
also have substantial scope for improvement, merely by providing more principled weights
for edit operations; in particular, the tree edit distance has the ability to increase or decrease
the estimated salience of diﬀerent structural features through such weights, and perhaps
better ﬁt the reality of human perception.
Although we have focused on Japanese, these models are all equally applicable to Chinese, subject to ﬁnding or constructing the requisite data sources, and may suggest approaches for other more general symbol systems which aren’t adequately indexed by theexisting body of research on image similarity search. Alphabetic scripts lack a tree-based
structure, though they may still use stroke sequences, and can easily be rendered as images,
indicating that at some of these metrics are transferrable more generally to other languages
with simpler scripts.

4.6

Conclusion

Visual similarity is assumed to be the basis for many kanji misrecognition errors, and soaccurate kanji similarity models have useful application in error correction and dictionary
search. In this chapter we ﬁrstly attempted to model similarity by using bag-of-radicals andimage diﬀerence approaches. In order to evaluate these metrics, we performed a large experiment which directly asked participants to rate the graphical similarity of random pairsof kanji. Our initial evaluation on this data showed best performance for the bag-of-radicals
metric, particularly when additional layout information was added, whereas the image distance metric performed poorly. We initially interpreted this to indicate the salience of radicals as sub-components. It remained clear that our metrics were still too noisy for downstream applications, and that the randomly chosen stimulus pairs followed too closely thenatural distribution of high-similarity pairs – that is, there were very few in the entire data
set. Since most applications rely heavily on high-similarity pairs, the data set was found toChapter 4: Orthographic similarity

91

be inappropriate for evaluating metrics for their intended purpose.
We addressed this issue by using human expert-selected similarity pairs, in the form ofpotentially confusable pairs borrowed from the White Rabbit Press JLPT 3 & 4 ﬂashcards.
We simultaneously considered two new metrics, both based on Ulrich Apel’s tree data forkanji: the stroke edit distance, and the tree edit distance. Both performed comparably
poorly to the L1 norm on the original data set, which required only that metrics correctly
order medium-to-low similarity pairs. However, on the new ﬂashcard data set, we founda reversal of these results. e radical level which had proved so appropriate for ordering
low-to-medium similarity pairs was simply too coarse to diﬀerentiate high-similarity pairs,
and was outperformed by the other metrics. In particular, stroke and tree edit distances
performed comparably best, followed closely by the L1 norm.
To determine how reproducible the ﬂashcard data set was, we performed a small experiment which took a number of stimulus kanji and for each kanji pooled the top-ranked
neighbours from each of our similarity metrics into a set of potentially confusable neighbours. We asked native or native-like speakers of Japanese to assess each neighbour pool andmark those neighbours which they considered truly confusable by learners. Since we had
few participants, we in turn pooled their aﬃrmative responses for each kanji presented. epooled responses agreed moderately with the ﬂashcard data, yielding a κ of 0.44, a result
better than any individual rater paired with the ﬂashcard data. is suggested that, although
individuals may diﬀer in their immediate conscious judgements, aggregated responses acrossgroups of raters hold the promise of matching more closely the human experience of similarity and visual confusion.
We now wish to apply these metrics to error modelling so that we may aid learners
in common tasks they perform. Since one of the most used tools in language learning isthe dictionary, the following chapter describes our attempts to enhance a dictionary withimproved usability and accessibility features. A key feature of this improved dictionary
is a search by similarity, which makes use of metrics from this chapter and incorporates
them into the error models required for lookup. Chapter 6 then takes these confusability
models one step further by incorporating them into automatic learner drills. e similarity
metrics from this chapter provide the basis for these new models of learner error and the
new applications that result.

Chapter 5
Extending the dictionary
In Chapter 3 we identiﬁed vocabulary acquisition as a crucial hurdle for language learners, and established the important role of dictionaries in supporting vocabulary learning,particularly in Japanese. We further argued that graphemic relationships modelling wasunderdeveloped in Japanese; these models were then developed in Chapter 4. is chapter
examines an individual approach to dictionary lookup, FOKS, and investigates how improvements to the method – including use of these new graphemic neighbour models –might better support both lookup and retention.
We begin in Section 5.1 with an extended background to the FOKS system, including
its original architecture and error models. Our work required re-implementation of FOKS;
signiﬁcant architectural changes from the original system are thus discussed. e remaining
three sections then describe improvements to the system.
Firstly, Section 5.2 examines the grapheme-phoneme alignment step in FOKS’s construction, crucial to providing accurate reading and alternation frequencies for its core error
modelling. We improved its accuracy and eﬃciency by adapting its unsupervised alignment
method into a semi-supervised method. Secondly, Section 5.3 investigates micro-structural
aspects of dictionary translations, with the goal of improving lookup and ultimately retention of the word once found. irdly, Section 5.4 combines FOKS’s phonemic errorcorrection with a novel graphemic error-correcting search, making use of the graphemic
neighbourhood modelling from Chapter 4. Indeed, the lack of such correction in FOKS
and other dictionaries was a signiﬁcant motivation for developing those models.
92

Chapter 5: Extending the dictionary

93

Finally, we look to log analysis in Section 5.5 to evaluate the eﬀect of these changes
where possible.

5.1

FOKS: the baselineIntelligent lookup
Our focus on FOKS requires a recap of its contribution to dictionary search in Japanese.
We can summarise the problems learners of Japanese (and native speakers) face when encountering a new word in the following way: words must be input into a computer bypronunciation, but the pronunciation is unknown when the word is unknown. For this reason, users normally have to fall back to a slower and more imprecise lookup method basedon visual analysis. is is the basic situation, but this description is incomplete.
ere are many situations when the learner has partial knowledge of the word they arelooking up. For example, the context in which the word occurs may give clues as to theword’s meaning – in fact the entire inferring-from-context strategy (Section 3.1) is basedon ﬂeshing out this partial knowledge over many exposures to a word. Equally commonly, akanji compound may be encountered where the learner knows something about each of the
kanji involved in the compound. Just as they might guess the meaning by combining themeanings of both elements, they can guess the pronunciation in the same way. In Japanese,such guesses are likely to be wrong for one of three reasons:
1. Each kanji may have several readings, yet a particular compound typically only has asingle valid reading. Choosing the correct reading for each kanji is diﬃcult.
2. When readings are combined into a whole, one or more combination eﬀects canoccur. For example, 日 nichi “sun” + 本 hoN “origin” combine to form 日本 nippoN “Japan”. e most prominent eﬀects are sequential voicing and sound euphony,which we discussed in more detail earlier in Section 2.3.
3. Some compounds have non-compositional readings, i.e. their pronunciation is notbased on any of their components, but is rather special to that compound. For example, 山車 dashi “festival ﬂoat” is non-compositional in this way.

94

Chapter 5: Extending the dictionary
When a learner guesses incorrectly and searches by this pronunciation in a typical dic-tionary, they will not ﬁnd the desired word, and will have to resort to slower and more complex searching. However, using the FOKS dictionary, they can search using the expected
pronunciation. Even if they choose the wrong reading, or mistakenly apply combinationeﬀects, FOKS will seamlessly correct the error and ﬁnd the word they were looking for
quickly and accurately from within the same interface. ey may equally use the interface
to intentionally search for words by an implausible but compositional reading; it will correct
for this as if it were simply an error.
For example, suppose the user wishes to look up 風邪 kaze “common cold”. He or she
may know the kanji 風 kaze/fū “wind”, and also 邪 yokoshima/ja “evil, wicked”, and thus
guess that the reading for 風邪 is kazeja, one possible combination of readings. However,the correct reading kaze is non-compositional. Figure 5.1 depicts the results of this searchin the FOKS interface. Despite the incorrect guess, FOKS still lists the target word withthe correct reading in its list of candidates for the guessed reading.

Figure 5.1: FOKS search results for the reading kazeja. e ﬁrst result 風邪 kaze “commoncold” is the desired word.

Chapter 5: Extending the dictionary

95

Architecture
FOKS is best termed a dictionary interface, rather than a dictionary resource; it serves as
an advanced indexing scheme constructed on top of an existing dictionary. is underlying
dictionary resource could be arbitrary, but in practice is a combination of one or more of thefreely available EDICT family of Japanese-English word-level dictionaries maintained by
the Electronic Dictionary Research and Development Group.1
is section provides an overview of the architecture of FOKS, as described by Bilac(2005), and thus shows how FOKS is able to recover from incorrect reading guesses. econstruction of FOKS has three main stages, as shown in Figure 5.2: grapheme-phoneme(GP) alignment, canonisation and reading generation. e simplest way to explain is tostep through FOKS’s construction.
Dictionary entries

Alignment
Segmented entry-reading pairs

Canonization

Alternation probabilities

Reading generation
Scored entry-reading pairs

Database

Figure 5.2: e architecture of the original FOKS interface.
For each entry in the dictionary which contains kanji, FOKS constructs an exhaustive list of plausible guesses for the word’s reading, scored by an estimate of their likelihood (or plausibility). In order to generate these readings, it needs to split words like
神社 jiNja “shrine” into their component segments by aligning word-reading pairs, a pro1

http://www.edrdg.org/

96

Chapter 5: Extending the dictionary

cess known as grapheme-phoneme alignment, using an unsupervised method described by
Baldwin and Tanaka (1999a). e result of this step is a series of GP-aligned words. Once
GP-aligned, our example becomes 神|社 ↔ jiN|ja.
ese alignments are used in two ways. Firstly, they are fed into a canonisation process
which recognises any reading alternations which have occurred. For example, 社’s reading
ja is recognised to be sha altered by sequential voicing. If r is the segment reading, rc is its
canonical reading, and s is its kanji form, then GP-alignment alone provides a frequency
distribution for Pr(r|s), whereas canonisation converts this into further distributions forboth Pr(r|rc , s) and Pr(rc |s).
Secondly, the alignment for each word is used as the basis for reading generation. Error
models use the frequency distributions from canonisation to generate a series of plausible
readings for each grapheme segment, usually in the form of an alternation from an existingreading. ese error models will be discussed further in the following section. Usually
the correct reading occurs naturally as one of the plausible readings suggested; in rare cases
where this does not occur, the correct reading is inserted into the generated list. Each word’s
readings are then weighted by the word’s corpus frequency Pr(w), pruned using a plausibility
threshold, and then stored in the database. From here, lookup is a simple matter of querying
against the plausible readings in the database – this is facilitated by a straightforward webinterface.
e overall lookup model for a word w given a reading query rw is as follows:

Pr(w|rw ) ∝ Pr(rw |w) Pr(w)= Pr(w) Pr(r1...n |s1...n )n∏≈ Pr(w)Pr(ri |si )

(5.1)

i

Firstly, Bayes rule shows the results can be ranked by Pr(rw |w) Pr(w); in this term,
Pr(w) is simply modelled by corpus frequency of words. We then break the reading andwords up into GP segments, and approximate with independence of GP-segments, giving
us the ﬁnal equation given in Equation 5.1. is equation relates plausible (mis)readings
for kanji with plausible (mis)readings for words.

Chapter 5: Extending the dictionary97

Error models
We suggested three main errors made by learners in our recap of lookup using FOKS:
inadequate choice of kanji reading, incorrect application of alternation rules for sequential
voicing and sound euphony, and non-compositional readings. A more thorough list of error
types encountered through extensive log analysis is discussed in Bilac (2005:25). Several ofthese extra error types are corrected by the original FOKS interface, including:
1. Palatalisation errors. For example, misreading 亜流 aryū “epigone” as arū.
2. Character/suﬃx co-occurrence errors. For example, misreading 激しい hageshī “violent”
as kibishī due to common suﬃx with 厳しい kibishī “severe”.
3. Insuﬃcient knowledge of proper nouns. Many proper nouns have diﬃcult or archaic
readings. For example, the word 上野 has at least 13 distinct readings as proper
nouns, including agano, ueno, and uwano for place names, as well as uehara, kamitzuke,
toshi for person names.
e last of these errors, insuﬃcient knowledge of proper nouns, is better described as
the problem of non-compositional readings.

Limitations
Language learners often have useful information they can use to constrain a query, but
are unable to express it in traditional electronic dictionaries. In particular, most only allow
partial but correct queries through wildcards. FOKS takes a unique approach amongst
Japanese dictionaries in allowing the user to express their partial knowledge – in this caseabout kanji pronunciation – in terms of a noisy query, from which it then attempts to recover
the original word. Despite success with this approach, the original system contained several
limitations.
A number of other errors were not corrected within FOKS, such as those due to:
1. Phonetic confusion of words or characters. Confusing a kanji with a homophone, andthen applying a diﬀerent reading borrowed from the homophone.

98

Chapter 5: Extending the dictionary
2. Graphemic similarity of words or characters. Confusing a kanji with a near-homograph,and then using a reading of the neighbour. For example, 闇 yami “darkness” being
misread as oN due to visual similarity with 音 oN “sound”.
3. Semantic similarity of words or characters. Confusing a word with a near-synonym, andthen using a reading of the synonym. For example, 火事 kaji “ﬁre” being misread as
kasai due to similarity with 火災 kasai “(disastrous) ﬁre”.
Phonetic, graphemic and semantic links or “associations” are all increasingly the focusof new dictionary interfaces, as discussed in Section 3.3. It is not surprising that they cause
errors in general, but it is at least unexpected that all three forms of proximity aﬀect searchby pronunciation in measurable ways. e magnitudes of these eﬀects are given ﬁgures in
Bilac et al.’s (2004) log analysis: 0.8% of errors were due to graphemic similarity, whereas
0.3% of errors were classiﬁed as either grapho-phonetic similarity, semantic similarity or
suﬃx co-occurrence errors. ese ﬁgures are simply lower bounds though, since only errors
corrected by existing error models were logged and analysed. More generally, such errors
would not have resulted in successful search, and thus would not be included in the log
analysis.
ese cases indicate that users sometimes have partial information about the word theywish to ﬁnd, which they express through their search (intentionally or otherwise). In thewell known tip-of-the-tongue phenomenon (Brown and McNeill 1966), the rememberer
can recall parts of the words pronunciation but not the entire word. Partial information
searches are aimed at allowing the learner to still ﬁnd their word, even in such cases. One
signiﬁcant limitation to FOKS is that it only allows partial knowledge to be expressed in
one manner – through pronunciation. Information about form or meaning is ignored. For
example, if any kanji are found in the query, FOKS limits the search to exact matches.
When FOKS does correct for misreadings, there is often lack of transparency in thecorrection method. For example, log analysis determined that 塵芥 chiriakuta “garbage”
was reached by the query chiNke, but the type of error made by the user was unclear even to
an expert (Bilac 2005:27).
Finally, an important limitation for faster experimentation is the build time for FOKS;
the grapheme-phoneme alignment cycle in particular is unsupervised, and takes a long timeChapter 5: Extending the dictionary

99

to complete, as does the oﬄine reading generation. If the longest word length is L, and there
are W words, then the time complexity for GP alignment is O(W 2 22L ). at is, it scales
quadratically with dictionary size, and exponentially with longest word size. is eﬀect isincreased as dictionaries increase their coverage and thus their size.

FOKS rebuilt
With many of these limitations in mind, we rebuilt FOKS with several signiﬁcantchanges. Firstly, many oﬄine steps such as reading generation were moved to become
online steps occurring at query-time. is had the beneﬁt of faster database rebuilds foreasier experimentation, but also that plausible misreadings generated by the system could
be easily reverse engineered to explain how they were reached.
Beyond these basic architectural changes, several other improvements were made to
FOKS, which form the focus of this chapter. In Section 5.2 we examine the graphemephoneme alignment algorithm and develop variants suitable for faster experimentation.
Section 5.3 then looks at so-called “microstructural” improvements to word translation to
improve usability, and extends coverage over place names. Section 5.4 then uses our newly
developed graphemic similarity models to provide error-correcting search by grapheme. We
conclude with log analysis of these changes in Section 5.5.

5.2

Grapheme-phoneme alignment

Overview
is section considers improvements to FOKS’s grapheme-phoneme alignment algorithm, however these improvements have potential for broader application beyond FOKS.
In our description and analysis of this problem, we situate them in this wider context.
e grapheme-phoneme (“GP”) alignment task aims to maximally segment the orthographic form of an utterance into morpho-phonemic units, and align these units to a phonetic transcription of the utterance. Maximal indicates the desire to segment grapheme
strings into the smallest meaningful units possible. Taking the English example word battleship and its phonetic transcription /bætlʃɪp/, one possible alignment is:

100

Chapter 5: Extending the dictionary
b a
b æ

tt le
t l

sh
ʃ

i p
ɪ p

Note that alignment in general is many-to-many. In the example above, tt aligns to/t/, le aligns to /l/ and sh aligns to /ʃ/. Equally it might be possible for some letters toalign to an empty string. is task is challenging for any language without a one-to-one
correspondence between individual graphemes and phonemes, as is the case with English(Zhang et al. 1999), Japanese (considering graphemes as kanji characters), and indeed mostlanguages with a pre-existing writing system.
Aside from FOKS, GP alignment is a prerequisite for many applications. For example, the alignment process, and its resulting aligned GP tuples, are a precursor to achieving
automated grapheme-to-phoneme mappings for text-to-speech systems such as MITALK(Allen et al. 1987), Festival (Black et al. 1999) and SONIC (Pellom and Hacioglu 2001).
Further uses include accented lexicon compression (Pagel et al. 1998), identiﬁcation of cognates (Kondrak 2003) and Japanese-English back-transliteration (Knight and Graehl 1998;
Bilac and Tanaka 2005).
ere are several successful approaches to Japanese GP alignment, notably the iterative
rule-based approach taken by Bilac et al. (1999), later followed by Baldwin and Tanaka’s(1999a) unsupervised statistical model based on TF-IDF. Although these models have highaccuracy, their iterative approach has a high computational cost, making them impracticalfor many real-world applications. For the statistical models, this is partially a consequence of
their strongly unsupervised nature. We thus explore the use of the EDICT and KANJIDIC
electronic dictionaries (Breen 1995) as means of constraining the alignment search spaceand reducing computational complexity.
is section examines in detail Baldwin and Tanaka’s (1999a) GP alignment method,
and alters it to achieve comparable alignment accuracy at a much lower computational cost.
To achieve this goal, we split the task of GP alignment into a pure alignment subtask andan okurigana detection subtask, and compare algorithm variants of pre-existing approaches
for both.
In Japanese, the GP-alignment problem is simpliﬁed somewhat by the convenience ofusing the syllabic kana script as the phonemic representation, though we will continue to useromanised forms in our examples. A simple example is given in Figure 5.3, where several

Chapter 5: Extending the dictionary101

possible alignments are shown. Kana are convenient in this context because they can occur
in both the grapheme and phoneme string, as in the suru suﬃx shown. Whereas kanji in thegrapheme string serve as wildcards, kana serve as inﬂexible pronunciations which constrain
the number of possible alignments.

感謝-suru

感謝-suru感-謝-suru

kaNsha-suru

kaN-sha-suru?

kaNsha-suru

Figure 5.3: e dictionary entry for 感謝する kaNshasuru “to give thanks, be thankful”,
with two of its potential alignments shown.

ere are four main word-formation eﬀects in Japanese which complicate alignment,each of which we discussed in Section 2.3. ey are: okurigana, sequential voicing, soundeuphony and grapheme gapping. e ﬁrst case, okurigana, describes inﬂectional suﬃxes
and how they may change. e remaining three cases describe eﬀects which can occur whenkanji form compounds. e last case, grapheme-gapping, needs little discussion: it occurs
very rarely (under 0.1% of the evaluation set) and is productive only in very limited forms,
as noted by Baldwin and Tanaka (1999b). It thus requires no special handling. In general
however, each of these eﬀects makes alignment more diﬃcult because they add variability
to kanji pronunciation.
In the remainder of this section, we ﬁrstly describe the existing algorithm in detail before
going on to describe our proposed improvements. Finally, we evaluate our results, discuss
the improvements and their re-integration back into FOKS.

Existing algorithm
Overview
We now examine the baseline GP-alignment algorithm used by FOKS, namely Baldwin and Tanaka’s (1999a) unsupervised iterative algorithm. A high-level depiction of this

102

Chapter 5: Extending the dictionary

grapheme-phoneme pairs

generatealignments
ambiguous alignments

apply linguistic
constraints
ambiguous alignments

solved alignments

TF-IDF scoring

iterative
disambiguation

solved alignments

Figure 5.4: e TF-IDF based alignment algorithm

algorithm is given in Figure 5.4. Firstly all potential segmentations and alignments for input
entries are created. In general, each entry may have potential segmentations and alignments
per segmentation numbering exponentially in the entry’s length. As in any alignment task
where two strings of length l and m respectively need to be aligned, there are 2lm possiblealignments before applying constraints (Brown et al. 1993).
Fortunately, some simple linguistic constraints avoid this worst-case number of alignments to consider. Alignments must be strictly linear, each grapheme must align to at least
one phoneme, and kana in the grapheme string must align exactly to their equivalents in thephoneme string. Further constraints used to prune entries include matching okurigana to
pre-clustered variants and forcing script boundaries (except kanji to hiragana boundaries)
to correspond to segment boundaries.
Based on the linguistic constraints, we can reasonably expect to have uniquely deter-Chapter 5: Extending the dictionary103

mined some number of alignments for any suﬃciently diverse data set.2 e uniquely determined alignments and the remaining ambiguous alignments are both used separately toseed frequency counts for the TF-IDF model.
TF-IDF is a family of models originally developed for IR tasks, combining the TF(term frequency) and IDF (inverse document frequency) heuristics (Salton and Buckley1988). In the GP alignment task, they mediate the tension between oversegmenting andundersegmenting. e TF value is largest for the most frequently occurring GP pair given
any grapheme; an oversegmented alignment produces rarer segments with lower frequency,
penalising the TF score. e IDF value on the other hand is largest for segments which
occur in a wide variety of contexts, and penalises undersegmenting.
TF-IDF Alignment
We use a modiﬁed version of the TF-IDF model which takes into account the diﬀeringlevel of conﬁdence we have in our frequency counts between solved (freqs ) and ambiguous(frequ ) alignments (Baldwin and Tanaka 2000). For each alignment, we count the occurrence of each grapheme segment ⟨g⟩, of each aligned grapheme-phoneme segment pair⟨g, p⟩, and of the same pair with one additional pair of context on either side ⟨g, p, c⟩. For
any frequency lookup, the ws and wu constants provide a weighting between informationfrom solved and ambiguous alignments:
wtf(x) = ws × freqs (x) + wu × frequ (x)(5.2)

To score a potential alignment, we calculate the TF and IDF scores for each graphemephoneme segment pair and multiply them together as in Equations 5.3-5.5. e score
for the whole alignment is the average of the scores for every pair which contains a kanji
character, since these are the non-trivial pairs. e constant α is intended as a smoothing
factor for the TF and IDF scores. It must be assigned such that 0 < α < wu ≤ ws .

TF(g, p) =2

wtf(⟨g, p⟩) − wu + α
wtf(⟨g⟩)(5.3)

Notable exceptions to this are dictionaries of 4-kanji proverbs, such as the 4JWORDS electronic dictionary, whose entries’ grapheme forms lack kana to help eliminate possible alignments.

104

Chapter 5: Extending the dictionary

IDF(g, p, c) = log(

wtf(⟨g, p⟩))wtf(⟨g, p, c⟩) − wu + αscore(g, p, c) = TF(g, p) × IDF(g, p, c)

(5.4)

(5.5)

Once all potential alignments have been scored, the highest-scoring alignment is chosen
to disambiguate its entry. Its counts are removed from the unsolved pool and added to thesolved pool, and algorithm reiterates with updated counts. In this way entries are iteratively
disambiguated until no more remain, and the algorithm is complete.
e iterative algorithm is eﬀective but extremely expensive, with two main components
to the cost. e ﬁrst is the number of potential alignments per entry, of which there areexponentially many. In particular, long entries with many kanji and no kana to constrain
them have prohibitively large numbers of possible alignments. ese cases bloat the numberof potential alignments to be rescored on each iteration so much that including them makes
our main algorithm infeasibly expensive: the longest few entries together have the same
number of potential alignments as the entire rest of the dictionary entries together. eonly way to tackle this component is to ﬁnd additional constraints which will reduce thenumber of alignments.
e second cost is in the scoring loop. Suppose there are n alignments pairs, eachwith p possible alignments. en the cost of the iterative rescoring loop is O(n2 p2 ). Even
having removed the problem cases above, if p is still high on average, the problem will prove
intractable for suitably large n. For example, the evaluation set used by Baldwin and Tanaka(1999a) has 5000 word-reading entries, yet the EDICT dictionary has over 170,000 entries
at the time of writing, representing an expected increase in computation time of three ordersof magnitude. Although this could be mitigated by simply breaking the input down into
smaller subsets for processing, it is desirable to process all the data in the same iterative loop,
since this gives greatest consistency of alignment.
Strategies to better constrain alignments and alternatives to iterative scoring form thebasis for our attempts at modifying the algorithm.

Chapter 5: Extending the dictionary105

Modiﬁed algorithm
e modiﬁed algorithm diverges from the unsupervised algorithm in three main respects. Firstly, we separate out okurigana handling into a separate step after alignment,
beneﬁting both eﬃciency and error measurement. Secondly, a reading model is introduced
based on the KANJIDIC electronic dictionary3 and is used to disambiguate the majorityof remaining cases before the TF-IDF model is reached. irdly, we provide a maximumalignment size cutoﬀ above which we use a simpliﬁed non-iterative alignment algorithm
which meets resource constraints for problem cases. We discuss these changes below.
Separating okurigana handlinge okurigana handling in the original algorithm involves pre-clustering okurigana alternates, and attempting to restrict alignments to match these alternates wherever possible.
Whilst this constraint does help reduce potential alignments, it also limits the applicationof the stronger constraint that script boundaries in the grapheme string must correspondto segment boundaries (i.e. every occurrence of a kanji–hiragana script boundary must be
considered as a potential okurigana site). If okurigana detection is left as a post-processingtask, we can strengthen this constraint to include all script boundaries, instead of omitting
kanji-to-hiragana boundaries. is in turn provides a larger gain than the original okurigana
constraint, since more entries are fully disambiguated.
e GP-alignment task is then split into two parts: a pure alignment task, which canbe carried out as per the original algorithm, and a separate okurigana detection task. isredesign also allows us to separately evaluate the error introduced during alignment, andthat introduced during okurigana detection, and thus allows us to experiment more freely
with possible models.
Short and long entries
Ultimately, any method which considers all possible alignments for a long entry will
not scale well, since potential alignments increase exponentially with input length. We can3

http://www.csse.monash.edu.au/~jwb/kanjidic.html

106

Chapter 5: Extending the dictionary

however extend the applicability of the algorithms considered by simply disambiguating
long entries in a non-iterative manner.
e number of potential alignments for an entry can be estimated directly from thenumber of consecutive kanji. Our approach is to simply count the number of consecutive
kanji in the grapheme string. If this number is above a given threshold, we delay alignment
until all the short entries have been aligned. We then use the richer statistical model to alignall the long entries in a single pass, without holding their potential alignments in memory.
Although long entries were not an issue in our evaluation set, a threshold set experimentally to 5 consecutive kanji worked well using the EDICT dictionary as input, wheresuch entries can prove diﬃcult.

Reading model
For the pure alignment task, we added an additional reading model which disambiguates
entries by eliminating alignments whose single kanji readings do not correspond to thosein the Kanjidic and KANJD212 electronic dictionaries. ese dictionaries list common
readings for all kanji in the JIS X 0208-1990 and JIS X 0212-1990 standards respectively,
covering 12154 kanji in total. Eﬀectively, we are applying the closed world assumption
and allowing only those alignment candidates for which each grapheme unit is associatedwith a known reading. Only in the instance of over-constraint, i.e. every GP alignment
containing at least one unattested reading for a grapheme unit, do we relax this constraint
over the overall alignment candidate space for the given grapheme string.
A simple example of disambiguation using the reading model is that of 一両 i-chi-ryou “one vehicle” as shown in Figure 5.5. Since only one of the potential alignments is compatible with the known readings, we then select it as the correct alignment. As an indicationof the eﬀectiveness of the reading model, our initial constraints uniquely determine 31.1%of the entries in the EDICT dictionary.4 e reading model disambiguates a further 60.6%of entries, eﬀectively decreasing the input to the iterative alignment algorithm by an orderof magnitude, to the remaining 8.3%.
4

http://www.csse.monash.edu.au/~jwb/edict.html

Chapter 5: Extending the dictionary

107

Potential alignments

一｜両i｜chi-ryo-u

一｜両i-chi｜ryo-u

一：i-chi, i-tsu, hi-to

一｜両i-chi-ryo｜u

一両
i-chi-ryo-u

両：ryo-u, te-ru, fu-ta-tsu

Kanjidic readings

Figure 5.5: Disambiguation using the reading model

Heuristic variants
We could continue to use the original TF-IDF model over the residue which is not disambiguated by the reading model, although the type of input has changed considerably after
passing through the reading model. Since the reading model is likely to fully disambiguate
any entry containing only single kanji segments, the only remaining ambiguous models arelikely to be those with solutions containing multi-kanji segments (which do not occur ineither KANJIDIC or KANJD212); an instance of a multi-kanji segment is our earlier example 風邪 kaze “common cold”. With this in mind, we compare the original TF-IDF
model (our baseline) with similar models using TF only, IDF only, or random selection tochoose which entry/alignment to disambiguate next.

Okurigana detection
We similarly wish to determine what form of okurigana detection and realignment
model is most appropriate. Since the majority of entries in the EDICT dictionary (ourmain experimental data set) which contain potential okurigana sites (i.e. kanji followed by
hiragana) do contain okurigana in some form, we use as our baseline the simple assumption that every such site is an instance of okurigana. In this manner, the baseline simply
removes every kanji-to-kana segment boundary. As a small enhancement, the boundary is

108

Chapter 5: Extending the dictionary

not removed if the tailing kana segment is one of the hiragana particles no, ga or ni, which
frequently occur alone.
We consider three alternative okurigana models to compare to our baseline, of increasing complexity and expected coverage. Firstly, the Kanjidic dictionary contains common
okurigana suﬃxes for some kanji with conjugating entries. us our ﬁrst model uses these
suﬃxes verbatim for okurigana detection. e coverage of okurigana suﬃxes in Kanjidic issomewhat patchy, so in our second model, in addition to Kanjidic suﬃxes, we also perform
a frequency count over all potential okurigana sites in the EDICT dictionary, and include
any occurrences above a set threshold as okurigana.
Finally, most instances of okurigana are due to verb conjugation. As well as taking
straight suﬃxes from the previous models, this ﬁnal model harvests verbs from EDICT.
Most verb entries in EDICT have a tag marking them as ichidan, godan or suru verbs.5 everb type and stem allow us to conjugate regular verbs variously, giving us a large numberof new okurigana suﬃxes not present in the previous models. In order to improve accuracy,
all three methods fall back to the baseline method if they do not detect any okurigana.

Evaluation
Having teased apart the alignment and okurigana detection algorithms, we are in aposition to separately evaluate their performance. Our test set for the combined task consistsof 5000 randomly chosen and manually aligned examples from EDICT, from which we thenseparated out an individual evaluation set for each sub-task.
Since we are also interested in eﬃciency, we provide execution time as measured byelapsed time on a Pentium 4 desktop PC. Our emphasis however is on the relative time
taken by diﬀerent algorithms rather than the exact time as measured.
In the following subsection we ﬁrst evaluate alignment and okurigana detection separately, then we evaluate okurigana detection, and ﬁnally we assess performance over the
combined task.
5

e tagset for EDICT verbs is larger than this, but the additional tags largely mark subclasses and exceptions of the three main classes, which we ignore for the sake of simplicity.

Chapter 5: Extending the dictionary

Iterative
Single-pass
Iterative + kanjidic

109
Random
47.8
47.3
94.4

TF IDF23.7 94.7
23.6 90.5
92.9 98.0

TF-IDF
93.4
90.8
97.9

Table 5.1: Alignment accuracy across models.

Iterative
Single-pass
Iterative + kanjidic

Random
0:10
0:10
0:12

TF
24:10
0:11
0:27

IDF TF-IDF
22:47
21:54
0:09
0:10
0:24
0:24

Table 5.2: Alignment execution time across models in minutes and seconds.

Alignment
We ﬁrst compare the accuracy of the three main alignment algorithm variants along withseveral scoring variants for each, as given in Table 5.1. e Iterative methods pre-populate
the frequency distributions with all potential alignments, and on each iteration they rescore
all potential alignments and resolve the single best. Since rescoring every potential alignment each iteration is expensive, we also provide a Single-pass variant for comparison. eSingle-pass method begins with the same alignment model, but uses only a single scoringround which determines the best alignment for every entry at once. e Iterative + kanjidic
method uses readings from kanjidic as soft constraints on potential alignments; in practicethis results in a signiﬁcant reduction in the number of ambiguous alignments. After someexperimentation, parameter values of 0.05 for α, and 2.5 for ws and wu were found to yield
the best results, and were hence used to generate the results we discuss here.
For each of the non-random heuristics, we expect that the iterative version will achieve
higher accuracy than the non-iterative version, since the statistical model is rebuilt each
iteration adding the best example from the last. As such, this represents a time/accuracy
trade-oﬀ, a fact conﬁrmed by our data (see Table 5.2). e gain (2% in the case of TF-IDF,
4% for IDF alone) comes at the cost of an order of magnitude larger execution time, which
also increases exponentially with the number of input entries.

110

Chapter 5: Extending the dictionary

In contrast, the Kanjidic model consistently achieves a very high accuracy regardlessof the heuristic chosen. A large number of entries are immediately disambiguated by the
Kanjidic model, thus initially improving accuracy and then facilitating use of more accurate
statistics in the iterative algorithm without signiﬁcant penalty to eﬃciency. We also expect
the Kanjidic model’s execution time to scale more moderately with the number of input entries than the original iterative algorithm, since a far lesser proportion of the entries require
iterative disambiguation.
Comparing the individual heuristics at this stage, a surprise is that the IDF heuristic attains equivalent results to the TF-IDF heuristic, suggesting that broad occurrence of⟨g, p⟩ pairs is a good indicator of their alignment probability. e TF heuristic in comparison performs worse than simply choosing randomly, suggesting that the proportion oftimes a grapheme occurs as the current ⟨g, p⟩ pair is a very poor indication of its alignment
probability. From a diﬀerent viewpoint, if TF guards against over-segmentation, then oversegmentations are not an issue in this task, where our goal is the maximal segmentation of
the GP pair.
Okurigana detection
We now compare the performance of our okurigana detection algorithms. All the algorithms we compare are linear in the size of the input and thus run in much less time than
the alignment phase, thus eﬃciency is not a signiﬁcant criteria in choosing between them.
e accuracy found by each model is shown in Table 5.3.
Model
Accuracy
Simple98.1%
Kanjidic
98.3%
Co-occurrence
97.7%
Verb conjugation
97.7%
Table 5.3: Okurigana detection accuracy across models

Interestingly, the simple baseline model which assumes that every potential case ofokurigana is okurigana performs extremely well, beaten only by the addition of the Kanjidic

Chapter 5: Extending the dictionary111

common okurigana stems. Adding more information to the model about valid okurigana
occurrences even reduces the accuracy slightly over our test data.
Rather than indicating blanket properties of these models, the results suggest propertiesof our testing data. Since it consists entirely of dictionary entries without the common
hiragana particles which would occur in open text, this greedy approach is very suitable,
and suﬀers few of the shortcomings which it would normally face.
In open text, we would consistently expect additional language features between lexical
items which would break the assumptions made by our simple model, and thus reduce its
accuracy dramatically. In contrast, the full verb conjugation model would then be expected
to perform best, since it has the most information to accurately detect cases of okurigana
even in the presence of other features.
Combined task
Selecting the two models which performed best on our test data, we can now evaluate
the pair on the combined task. For the alignment subtask, the IDF heuristic with Kanjidic
was used. For the okurigana detection subtask, the simple algorithm is used. e results
are shown in Table 5.4.
StatusCount Percentage
Correct
4809
96.2%
Incorrect
191
3.8%→ Gapping6
0.1%→ Alignment
163
3.3%→ Okurigana
22
0.4%
Table 5.4: Best model accuracy for the combined task

A ﬁnal accuracy of 96.2% was achieved, with the errors caused mostly in the alignment
subtask. As predicted, grapheme gapping was a source of errors only in a small percentage
of cases, justifying its exclusion from our model. is level of accuracy is equivalent to thatof earlier models, yet it has been achieved with a much lower computational cost. Examples
of incorrect alignment are given in Figure 5.6 below.

112

Chapter 5: Extending the dictionarya.

b.

Output 挾｜撃｜chi hasa｜miu｜chiCorrect 挾｜撃｜chi hasami｜u｜chi“pincer attack”
Output 赤-N｜坊
akaN｜bōCorrect 赤｜N｜坊 aka｜N｜bō“baby”

Figure 5.6: Examples of incorrect alignment in the combined task

Example (b) shows a typical alignment error, where one kanji has been attributed part of
the reading of another. Example (b) on the other hand gives an error in okurigana detection,
where the N kana is erroneously detected as an okurigana suﬃx of the 赤 kanji.

Improved alignment
We have decomposed the GP alignment task into an alignment subtask and an okurigana detection subtask, and explored various algorithm variants for use in both. In particular,
the iterative IDF heuristic with a Kanjidic reading model provided the best accuracy in signiﬁcantly less time than the original algorithm. For the okurigana detection subtask, a
simple model outperformed more complicated models of conjugation due to peculiarities
of dictionary entries as input to alignment.
e modiﬁed algorithm was suitable for use with the FOKS system, and was thus
adopted as part of the build process for the new FOKS interface. Ideally, this work would
have applicability to open text as well as dictionary interfaces. However, the basic unsupervised method relies heavily on getting a representative sample of readings from the
input before attempting alignment. One way to circumvent this would be to bootstrap the
method with alignments from an existing representative sample, say the EDICT dictionary,and then use the bootstrapped algorithm to align new pairs.
Okurigana detection remains the harder problem, for tasks which require it. e verbconjugation model, despite its relatively poor performance for dictionary entries, suggests
itself as the most fruitful approach to accurate detection for open text, and could easily be
extended. In particular, the addition of conjugation suﬃxes of high-frequency irregularChapter 5: Extending the dictionary

113

verbs would be a straightforward way to boost accuracy.

5.3

Usability enhancements

is section describes three basic enhancements to the dictionary, namely: the additionof a huge library of place names, and their display; the separation of the senses of polysemous
kanji and words; and the ability to explain to a user how a query worked, and how the correct
reading is structured. We discuss each of these in turn.

Place names and homographs
As part of error analysis of FOKS, Bilac et al. (2004) found that insuﬃcient knowledge of proper nouns was a common error type, as discussed in Section 5.1. FOKS isespecially useful for these proper nouns, since their pronunciation is idiosyncratic. However, FOKS relies on ENAMDICT for place names, which has two disadvantages. Firstly,
ENAMDICT has limited coverage of place names in Japan. Secondly, it gives no cues as
to relationships between places, only giving each place name a transliteration as its gloss.
We address both of these problems by constructing a simple gazetteer resource fromdata mined from Japan Post.6 is resource provides a large number of place names not in
ENAMDICT, but also provides some hierarchical structure which we can use to distinguishbetween the many homographs encountered in this exercise, structure which ENAMDICT
does not provide. 114591 place names were mined; of these, roughly 69% had uniquewritten forms, whereas 31% did not. Figure 5.7 shows the distribution of homography overthe 79139 unique written forms mined.
e extremes of this distribution also suggest the limits of this data set. e most used
place name found was 本町, with 315 places using this name, taking pronunciations hoNchō,
hoNmachi or motomachi. However, this is better termed a suﬃx than a place name in its own
right. More generally, there are a number of valid but high-frequency place names, such as
上野, with 76 matches. In reality, the senses of such place names – that is, the places they
actually refer to – themselves vary in frequency. For 上野, the most prominent amongst
6

e Japan Post Gazetteer: http://www.csse.unimelb.edu.au/~lljy/datasets/#gazetteer

# places
(cum)
79139
9248
4663
3097
2246
1747
1414
1170
1011
871
725
631
565
510
456
403
360
323
298
279
248
230
211
198
183
166
158
145
140
136
131
122
116
108
103
98
95
92
87
83
81
76
72
70
66
62
56
54
51
50
48
47
45
43
41
40
37
35
33
31
30
28
27
26
25
23
22
20
19
18
17
16
14
13
12
11
10
9
8

Chapter 5: Extending the dictionary
Homography in Japanese place names
100000# place names (cumulative)

ces
69891
4585
1566
851
499
333
244
159
140
146
94
66
55
54
53
43
37
25
19
31
18
19
13
15
17
8
13
5
4
5
9
6
8
5
5
3
3
5
4
2
5
4
2
4
4
6
2
3
1
2
1
2
2
2
1
3
2
2
2
1
2
1
1
1
2
1
2
1
1
1
1
2
1
1
1
1
1
1
1

114

10000

1000

100

10

1

1

18

35

53

83

315

# homographs

Figure 5.7: e distribution of place-name homography in Japanese. e vertical axis shows
how many place names had at least n homographs, out of the 79139 unique place names
encountered.

these senses is likely to be the Ueno area of Tokyo. FOKS currently lacks a useful way ofdetermining and displaying the relative importance of diﬀerent place name senses. Instead,
we simply group these names by region, so that the user can ﬁnd the place they were looking
for provided they know its broad region. e more diﬃcult task of determining the mostprominent place name senses remains as future work. For each of these place names, we
generated an automatic transliteration using Kakasi7 to serve as a gloss.
Although the inclusion of this larger number of place names will allow the user to ﬁndthem in the dictionary, it could also pollute the search results for common words. is is
avoided in FOKS in two ways. Firstly, FOKS is primarily an indexing method for kanji
compounds. Even in the case of 本町 above with 315 matching places, only a single entry
is placed in the search results for a matching query. Secondly, users can ﬁlter by the type ofquery they are performing, or choose a default ﬁlter to apply to all queries. is provides aneasy way to either focus on or exclude place names from results as desired. Search results
7

http://kakasi.namazu.org/

Chapter 5: Extending the dictionary115

aside, we have also made improvements to a word’s translation page, which we now discuss.

Sense separation
Both the new and old FOKS interfaces are targeted at achieving successful lookup ofa word, in particular words containing kanji. However, the large amount of homography
in Japanese means that the same word or compound might be used for several diﬀerent
purposes, and the two interfaces diﬀer in how they present this information to the user. In
the old interface, a gloss is shown for each word whose surface form matches exactly thatof the successful query. When a word has multiple readings for the same gloss, this leads
to repetition of the gloss. If many proper nouns are included in the translation results, they
can pollute the results, depending on the purpose of the user’s search.
e new interface takes a diﬀerent approach in allowing the user to visualise the complex
relationship between surface form, readings and senses. For purposes of comparison, Figure 5.8 provides screenshots of both systems for the translation of compound 下手. Firstly,
the three main categories of sense – general word, person/company name and place name –are clearly demarcated in separate sections. is allows the user to skim to the section theycare about. e general word section in turn separates out reading-senses (senses which
occur with a speciﬁc reading), and groups them by pronunciation. For example, shitate andshitade are both listed as readings of the sense “humble position”. e two sections of proper
nouns both list not translations but transliterations (i.e. romanisations) of the proper nouns.
e section for place names also provides a hierarchical display of the place name, showing
country, prefecture, ward and town; in this case 下手 shimode is located in Kagoshima Prefecture. Clicking on any of the levels in the place hierarchy makes a Google Maps query,
locating the place quickly and conveniently.

Query and reading explanation
e original form of FOKS search bases itself around plausible misreadings of kanji
and kanji compounds, and indeed reverse engineers the most prominent word-formationeﬀects in order to determine what a plausible reading would look like. However, there is alack of transparency for users in how FOKS works, and why it gave them a particular search

116

Chapter 5: Extending the dictionary

Figure 5.8: e old and new translation pages shown for the compound 下手.

Chapter 5: Extending the dictionary

117

result. Since the new FOKS architecture generates the misreadings at query-time, it canalso reverse-engineer queries to determine the relationship between the misreading and thetarget.

Figure 5.9: FOKS explaining the relationship between the query tokyo and the word
東京 Tōkyō.

Figure 5.9 gives an example for a common erroneous query, searching for 東京 Tōkyō
using the reading tokyo; this is a misreading because both vowels should be long, but are
instead short vowels in the query. e explanation ﬁrstly segments the reading into perkanji readings, then explains the erroneous reading for each kanji as a correct base readingchanged due to a vowel length error. e explanation also notes that the reading is incorrect,
lists correct readings for the compound, and oﬀers explanations for how the correct readings
are derived. is last point is especially useful for learners who may be unsure how thereading of a long word can be attributed to readings of its parts.
In some cases, such as 海豚 iruka “dolphin”, the correct reading is non-compositional. In

118

Chapter 5: Extending the dictionary

these cases, the explanation cannot provide a segmented per-kanji reading; it instead simply
states that the reading is correct but non-compositional, and thus special to this compound.
To recap our usability enhancements, we have extended the dictionary entries with alarge number of proper nouns in the form of place names, yet managed these additional
candidates through the use of search ﬁlters and appropriate translation structure to avoid
them burdening search for common words. We have provided an augmented word translation page, which organises word readings and senses in a manner which both oﬀers the user
all the information about the word without overwhelming them. Finally, we have provided
transparency to the search, by explaining to users how queries are reached, and providingthem with the tools to better study and understand word formation principles in Japanese.
Whilst we oﬀer no strong evaluation for these changes, we make the case that they improvethe value of the dictionary as a study tool, and may also improve retention of looked-up
words.

5.4

Intelligent grapheme search

Overview
In Section 5.1 we gave an overview of FOKS, and discussed the shortcomings of theold system. ey came in two forms: errors in searches by reading which could not becorrected, and searches by grapheme which were limited to exact matches. Having discussed
and evaluated carefully various models of graphemic similarity, we are now in a position toprovide this form of error correction to FOKS searches.
When we originally developed our graphemic neighbourhood models, we intended toprovide correction for mistaken searches by reading. Our motivation came from Bilac et al.’s(2004) log analysis of FOKS, which picked up a small number of these cases. Although thenumber itself was low, the log data itself was highly constrained: it only contained successful searches through FOKS, which only corrected for reading errors. us, to appear in
the logs at all, such grapheme-confusion cases needed to coincide with some other form of
reading error which FOKS did support. We wouldn’t normally expect grapheme-confusion
and other forms of reading error to coincide, so the true prevalence of this error type mightChapter 5: Extending the dictionary119

be signiﬁcant. However, allowing search for a word using a reading of a near-homographrepresents a trade-oﬀ, since the many readings available for any kanji would swell the potential candidate list for any search enormously. An alternative view of this problem is that thelink between a compound’s reading and that of near-homographs is tenuous; attempting tocorrect such errors forces us to cast our net too wide. Either the results will appear, but will
be lowly ranked, or they will displace more frequent error types, and thus decrease search
performance.
For these reasons, we extended search by grapheme – which had been exact match only –into an error-correcting search, with the potential to both correct errors based on graphemic
similarity and also to allow intentional search of unknown characters, using known neighbours. is section describes the details of this new error-correction model.

Overall modele broad probability model for looking up words based on similar kanji is identical to
the FOKS model for search based on readings, save that we substitute readings for kanji in
our query. A unigram approximation leads us to Equation 5.6 below, where q is the querygiven by the user, and w is the desired word:

Pr(w|q) ∝ Pr(w) Pr(q|w)∏= Pr(w)Pr(qi |w, q0 . . . qi−1 )i≈ Pr(w)∏Pr(qi |wi )

(5.6)

i

e ﬁnal line of Equation 5.6 requires two models to be supplied. e ﬁrst, Pr(w), is
the probability that a word will be looked up. Here we approximate using corpus frequency
over the Nikkei newspaper data, acknowledging that a newspaper corpus may be skewed
diﬀerently to learner data. e second model is our confusion model Pr(qi |wi ), interpreted
either as the probability of confusing kanji wi with kanji qi , or of the user intentionallyselecting qi to query for wi . It is this model which we now focus on.

120

Chapter 5: Extending the dictionary

Confusion model
Our confusion model must account for two main factors. Firstly, it must be based on
the visual similarity of two characters qi and wi , which we address by use of one of ourdistance metrics. Secondly, we expect that users will tend to confuse unknown characterswith characters that they are already familiar with. is is a reasonable assumption since
the only characters they can input are those they already know. We thus propose a genericconfusion model based a similarity measure between kanji:
Pr(qi )s(qi , wi )j Pr(qi,j )s(qi,j , wi )Pr(qi |wi ) ≈ ∑(5.7)

e confusion model uses a similarity function s(qi , wi ) and a kanji frequency model Pr(qi )to determine the relative probability of confusing wi with qi amongst other candidates. We
convert the desired distance metric d into s according to s(x, y) = 1 − d(x, y) if the range
1
if the range of d is [0, ∞).
of d is [0, 1], or s(x, y) = 1+d(x,y)

In order to maximise the accessibility of this form of search, we must ﬁnd the appropriate
trade-oﬀ between providing enough candidates and limiting the noise in the candidates
present. We use a thresholding method borrowed from Clark and Curran (2004), where
our threshold is set as a proportion of the ﬁrst candidate’s score. For example, using 0.9 as
our threshold, if ﬁrst candidate has a similarity score of 0.7 with the target kanji, we would
then accept any neighbours with a similarity greater than 0.63. Using the dstroke metric witha ratio of 0.9, there are on average 2.65 neighbours for each kanji in the jōyō character set.
Search by similar grapheme has an advantage to search by word reading: reading results
are naturally ambiguous due to homophony in Japanese, and attempts to perform error
correction may interfere with exact matches in the results ranking. In contrast, graphemebased search may have only one exact match, so additional secondary candidates need not
be in direct competition with existing search practices.
Finally, we can consider the theoretical accessibility improvement of this form of search.
Let us assume that learners study kanji in frequency order. For each kanji learned, one ormore high-similarity neighbours also become accessible. Taking all pairings of kanji within
the JIS X 0208-1990 character set, using the dstroke metric with a cutoﬀ ratio of 0.9, we
get the accessibility curve found in Figure 5.10. Our baseline is a single kanji accessible

Chapter 5: Extending the dictionary

121

Accessibility of similarity search# kanji accessible

6000

4500

3000

1500

0

50 400 800 1250 1800 2350 2900 3450 4000 4550
# kanji known

baseline

accessible by search

Figure 5.10: e potential accessibility improvement of kanji similarity searchfor each kanji learned. Given that the actual number of usable neighbours is small and our
precision within top-5 candidates of 0.228, we will need to expose the user to a larger set ofcandidates in order to get this level of improvement.
In order to determine how this form of search is actually used by learners, we now look
to log analysis.

5.5

Log analysis

is chapter has described a broad series of improvements to the FOKS dictionary,extending it with the new intelligent grapheme search, and providing a large number of usability enhancements. Many of these enhancements are diﬃcult to evaluate directly. Dictionary search takes relatively unconstrained and free-form input from users, and this canbe hard to interpret in the resulting logs. We address this problem in Chapter 6 in a moreconstrained testing application, where we provoke errors from our users and can thus eval-

122

Chapter 5: Extending the dictionary
Language
Japanese
EnglishChineseTurkish
Other
Total

Visits
42798
6781
1164
765
2301
53809

Cumulative time onsite (hours)
79.5% 1785
66.1%
12.6%
623
23.0%
2.2%
148
5.5%
1.4%
20
0.7%
4.3%
126
4.7%
100.0% 2702
100.0%

Table 5.5: FOKS usage as reported by Google Analytics in the period from October 2007
to March 2009, broken down by browser reported language.
uate our error models more rigourously. Nonetheless, the broad usage patterns of FOKS
can still be examined. e new system with grapheme error-correction was implementedin October 2007, and our results are based on logs taken from then until late March 2009.
Over this period, Google Analytics was used to measure site traﬃc and get a better understanding of user location and language background. e majority of queries originatedfrom within Japan, which is unsurprising since both native speakers and advanced learners are likely to be based there. Table 5.5 shows the breakdown of site usage by browser
reported language. Note that the browser reported language may diﬀer from a user’s actual ﬁrst language; for example a native German speaker may be running Japanese browser
software if they are an advanced learner, or working in a Japanese computing environment.
Nonetheless, it gives us an indication of usage patterns. 80% of visits are Japanese-language,
perhaps indicating a large pool of native-speakers as users, with English language visits at
13%. Smaller pools of Chinese and Turkish speakers are also signiﬁcant users. However,when measured by time actually spent on the site, the share given to Japanese languageusers reduces to 66%, since they spend less time on average on the site than other language
groups.
We now turn to query logs. Any interpretation of such logs bases itself on a model ofuser behaviour. Our basic assumptions are that a user is looking for a particular word whosekanji form is available, known or recognisable to them. When a user performs a search, they
are thus presented with a list of scored word-reading results, each with a “Translate” button
next to it. When a user clicks on this button, FOKS captures their decision and displays adetailed translation of the word. is is presumed to be the word the user intended to ﬁnd,

Chapter 5: Extending the dictionary

123

Full query Partial query
TotalIntelligent Simple
By grapheme
997 16756
81018
98771
By phoneme
754
4262
22912
27928
Total
1751 21018
103930 126699
Table 5.6: Number of FOKS searches logged between October 2007 and March 2009,
arranged by search type.
and thus considered a successful search. We call such queries full queries, and those where
the user does not select any word for translation partial queries, after Bilac et al. (2004).
Partial queries give us little useful information, whereas full queries allow us to compare thequery with the word found and determine if any error-correction was used.
is model represents only one possible series of user interactions, but we assume it is thedominant user behaviour. Other behaviours include: clicking on translations of unknown
words mid-search out of curiosity, whether or not their desired word is present in the results;
looking for a word of known pronunciation but unknown meaning, thus clicking on severaltranslations to determine which matches the known meaning; searching to determine thepronunciation of a word of known form; searching to determine the form of a word of
known pronunciation; and others. Distinguishing between these behaviours would be aninteresting exercise, but is not the focus of this thesis, hence our assumption of a dominantbehaviour.
Using these assumptions, basic search statistics can be compiled, as shown in Table 5.6.
Roughly 29% of grapheme searches and 30% of phoneme searches are full queries, theremainder are partial queries. Of the full queries, we divided them into queries which werecorrect (i.e. exact-matches), and those which contained some error which our error models
corrected. e former type of query we call “simple”, the latter “intelligent”. Focusing on thefull queries, searches by grapheme are over 5 times more prevalent than those by phoneme,
perhaps indicating the common practice of copying and pasting an unknown word into thedictionary. However, they enjoy a comparable share of the intelligent searches.
Since one feature of the rebuilt architecture was the ability to reverse-engineer the relationship between a query by reading and a word, we can give an automated breakdownof the prevalence of reading error types. is breakdown is given in Table 5.7. Inadequate

124

Chapter 5: Extending the dictionary
Error type

Frequency

Choice of reading
547 72.0%
Vowel length
92 12.1%
Non-compositional reading 59
7.8%
Sequential voicing
40
5.3%Sound euphony
13
1.7%
Palatalisation
9
1.2%
Total
760 100.0%

ExampleQuerynorirutokyo
umibutahayahayafukukiseNko

Word
乗る noru “to get on”
東京 tōkyō “Tokyo”
海豚 iruka “dolphin”
早々 hayabaya “early”
復帰 fuk-ki “return”
選挙 seNkyo “election”

Table 5.7: Automatically determined error types for intelligent queries by reading.

choice of reading the most prevalent error type, accounting for 72% of errors. Vowel length
errors account for a further 12%, followed by the other error types in smaller proportions.
is drop-oﬀ diﬀers somewhat from Bilac et al.’s (2004) earlier log analysis of the original
system, in that we encounter a far smaller proportion of gemination errors, voicing errors
and errors due to non-compositional readings. is could be due to subtle changes in search
result ranking of the new system, or even in the demographic of its users. Nonetheless, eachof the broad error types is successfully corrected for by the new system.
Of the 997 grapheme-similarity searches, the forms of analysis open to us are limited.
We can however test the assumption inherent in our error model, namely that kanji will be
mistaken for their more frequent but visually similar neighbours. Looking at every intelligent grapheme query, and every target kanji t and query kanji q which diﬀer, let us measure). is log-ratio indicates the relative frequency of kanji errors compared to thelog( Pr(q)Pr(t)
kanji they are derived from. If strongly positive, this would suggest either that people confuse kanji for their more frequent neighbours or that people use more frequent neighbours
to query for rare kanji. If strongly negative, the result would be problematic: it would seemto suggest that people use rare kanji to query for common ones, a highly unlikely scenario.
Instead, we would surmise either that unexpected confusion errors were occurring in selecting the appropriate search result, or alternatively that the corpus we use to measure kanji
frequencies diﬀers signiﬁcantly from the order in which learners acquire kanji.
Plotting a histogram of the log ratio, Figure 5.11 shows that the distribution is remarkably even, showing a very slight negative log-ratio on average. is suggests that, on

125

n
0
7
22
94
441
401
104
19
2
1

Grapheme correction query-target likelihood ratio
500

Number of corrections

Log diff
0.8
4
0
6
2
2
6
0
4
.8

Chapter 5: Extending the dictionary

375

250

125

0

-10.8 -8.4 -6.0 -3.6 -1.2

1.2

3.6

6.0

8.4 10.8

Log of query/target likelihood ratio

Figure 5.11: Histogram of log query/target likelihood ratio for all corrected grapheme
queries, taken on a per-kanji basis.

126

Chapter 5: Extending the dictionary

average, kanji are mistaken for visual neighbours of comparable but not necessarily higher
frequency, thus providing evidence for neither of the outcomes discussed above. Instead, aplausible explanation could be that learners tend to confuse kanji pairs on the periphery of
their knowledge, rather than mistaking rare kanji for higher frequency known ones.
An alternative hypothesis is that users are distracted by the additional search results,
which anecdotally seem to contain individually frequent kanji in rarer combinations, andmay thus select a word other than their original target for translation. For example, a searchfor 海豚 iruka “dolphin” also yields the surname 海家 Umiie in the search results, which is arare composition of the highly frequent kanji 海 and 家. Combinations such as this might
be interesting enough to distract the user from their original search. Whilst this behaviour
is not technically an error, it would break our simplifying assumption that the ﬁrst word auser selects in search results was the original word used to formulate the query. is casecould potentially be investigated through more sophisticated analysis of log behaviour, since
it should be identiﬁable by the user selecting multiple query results for translation, one ofwhich may be an exact match for the original word. Such analysis is beyond the scope of
this thesis; we leave it for future work.

5.6

Conclusion

If learning words is the main obstacle to linguistic competency, then the dictionary is
the gateway to this competency, since it is the primary resource for unlocking the meaning
behind otherwise foreign words. In this chapter, we examined the FOKS dictionary, withthe aim of making it a better resource for learners in their autonomous study. We took aholistic approach, taking apart the dictionary and rebuilding it with improved graphemephoneme alignment, improved place-name coverage, better word translation, explanationsof word readings, and ﬁnally, a new form of search which allows users to substitute unknown
kanji for their visually neighbours in queries. Our log analysis shows that the number ofintelligent graphemic queries was higher than that of intelligent phonetic queries, indicating
the adoption of this form of search by the existing user base.
ere are many directions which future research could take in improving further uponthis form of search. Log data accumulating from intelligent graphemic queries can be storedChapter 5: Extending the dictionary

127

and used as a means to evaluate improvements to graphemic error correction. Furthermore,such data also serves as yet another means of evaluating orthographic neighbourhood models, which still contain substantial noise and hence potential for improvement. Our automatically generated explanations for both correct and plausibly incorrect readings of kanji
compounds could be extended to graphemic errors, identifying the strokes or components
which diﬀer between the two characters and highlighting the diﬀerences between them.
ere is also potential for other forms of graphemic relationships such as subsumption to
be used as graphemic search methods. For example, 動 could be used to search for 働, notbased on their similarity, but based on its strict membership within the latter kanji. Such asearch method has already been trialled by Jin (2008), but with further work the comparison
between graphemic search methods could help us to better understand what kinds of partial
information are most naturally available and most easily expressible in search queries.
Ultimately, at the core of the new FOKS implementation we have discussed lies two
main error models: a model of plausible mispronunciation, and a model of plausible misrecognition. Yet the use of these models need not be limited to the dictionary. We now
continue by applying these models of plausible error in language testing domain, in orderthat students can better self-evaluate their study.

Chapter 6

Testing and drilling

In the previous chapter (Chapter 5), we examined the eﬀect of augmenting an existing dictionary with the graphemic error models we developed earlier (Chapter 4). In thischapter, we examine the transferability of both phonemic and graphemic error models tothe domain of language testing and drilling, using a new system called Kanji Tester.

We begin in Section 6.1 by discussing why testing is such an attractive application space
for the graphemic and phonetic error models we used in search. We then propose criteriawith which to situate Kanji Tester amongst other forms of testing. Since Kanji Tester ismodelled after the Japanese Language Proﬁciency Test ( JLPT), Section 6.2 describes the
JLPT in more detail and justiﬁes our interest in emulating it with our automated tests.
With this background complete, Section 6.3 sets out describing the Kanji Tester system indetail, from the perspective of both its use and its implementation. It discusses our usermodelling, from the granularity of modelling chosen to the algorithms used for update andfor question generation. Section 6.4 then evaluates Kanji Tester through analysis of its usage
logs, exploring user demographics, user ability, power users and the overall eﬀectiveness of
our error models. Finally, we conclude in Section 6.6 with our ﬁndings.
128

Chapter 6: Testing and drilling

6.1

129

From search to testing

Testing as an application space
We saw in Chapter 3 that tools for learning vocabulary can basically be divided into
two forms: reading aids and tests. ese mirror the two ways in which vocabulary can belearned, implicitly or explicitly. So far, we have focused on dictionaries, since they are thesimplest reading aid and ﬁrst point of contact for learners. We now complete our study byconsidering language testing.
ere are several strong reasons to choose testing as our next application area. Without
exception, all forms of explicit vocabulary study focus heavily on testing. is is because
as learners we are poor at self-evaluation; by testing ourselves we can objectively discover
the limits of our knowledge, and thus begin to expand them. Testing also closes anotherloop: motivation. A strong motivation may provide the impetus for early study, but without
evidence the study is paying oﬀ, it may wane. Testing provides this evidence.
Aside from its heavy use in language learning, testing is also an attractive secondary application area for the error models we generated for search. In search, the learner provides
us with their response to an unknown stimulus, and our task is to recover the original stimulus word. Since user queries are noisy, we are often unable to recover the original word.
Such cases can be attributed to one of three issues:
1. Insuﬃcient user knowledge: e user did not know enough accurate information about
the word to formulate a useful query.
2. Poor modelling of error prevalence: e search results contained the desired word, but
the rank was too low for the user to ﬁnd it.
3. Poor coverage of errors: e search results did not contain the desired word, despite acorrect query formulation by the user.
Unfortunately, we can’t distinguish between the ﬁrst error type, which can’t be reduced, andthe latter two, which can. In testing however, the system generating the test provides thestimulus to which the user must respond. is more constrained form of interaction means

130

Chapter 6: Testing and drillingthat user responses can be used directly to evaluate our error models and thereby form abasis for their improvement.

e ideal test
It makes little sense to investigate language testing without considering what makes atest worthwhile. We discussed in Section 3.4 the two main quality attributes of tests, namely
validity and reliability, and these are certainly desirable properties for any test. However,whilst they are useful for comparing two tests with the same aims, they oﬀer little guidance
as to what to test in the ﬁrst place. For example, many tests from ﬂashcards to large scaleaccredited language testing can achieve high validity and high reliability, albeit measuring
fundamentally diﬀerent properties of the learner.
In order to make sense of the variety of tests used in language study, we instead propose
two scales along which we arrange these and other tests measuring language proﬁciency.
e ﬁrst is the scope of the test: does it attempt to measure all aspects of proﬁciency, alimited subset, or simply basic knowledge of a few words? e second scale we call the test’s
availability, a measure which encapsulates both its accessibility and any obstacles to taking
it: can the test be taken on demand, or only rarely? does it take a long time, or cost money?
Figure 6.1 arranges several forms of tests along these scales.
We justify these scales by revisiting the actual purpose of testing. Testing is used toprovide information supporting decision making. In the context of language learning, suchdecisions range from do I need to keep studying this word?, to the much broader what progress
have I made? and what study methods or classes were most eﬀective for me? Simpler decisions
can already be made quickly and eﬀectively with the use of basic tools such as ﬂashcards. It
is the more complex decisions which require much stronger supporting information. Largescale formal tests of proﬁciency, such as Japanese Language Proﬁciency Test ( JLPT), aredesigned with wide scope to meet this need, but currently this scope comes at a great costto availability. For many learners, the information provided by this form of test comes too
infrequently to usefully inform their study.
Imagine an oracle which, when asked, could accurately and instantly determine the proﬁciency of a learner, across whichever dimensions this proﬁciency is best measured. Such anChapter 6: Testing and drilling131

ideal test

JLPT

scope

class test

proposed
systemflashcards

availability
Figure 6.1: e various types of Japanese proﬁciency testing as they vary in scope and availability. e dotted line represents the current state-of-the-art.
oracle would reduce or eliminate the need for formal testing, and would itself be considered
the ideal test. is is indeed the underlying goal of Intelligent Tutoring Systems, where theuser’s study of content material informs a user model which in turn determines what will
be presented to them next based on their current proﬁciency (Shute and Psotka 1994:10).
From another perspective, these platforms oﬀer a form of continuous testing where learning and testing are either combined into a single activity or rapidly alternated between. emore the learner uses the system, the more information it has to use to aid the learner.
Without the pre-existing user data which such a platform can provide our ideal test
would share much with existing broad-scope manually-constructed tests such as the JLPT.
e main problem with such tests currently is that they cannot be taken more than once
without damaging their validity; once a learner has seen a test, their performance on subsequent attempts at the same test may be aﬀected as much by their memory as by their ability. For this reason substantive tests are usually rewritten for every use by expert language
instructors, at signiﬁcant time and cost. e transition to computer-based testing allows
postponing of this problem through the manual construction of large question banks, but
the fundamental issue remains: if individuals are allowed to test repeatedly, validity suﬀers.

132

Chapter 6: Testing and drilling

In contrast, today’s automatically generated tests have high availability, but are universallyprimitive in the forms of questions they can oﬀer, which in turn reduces their validity inassessing language proﬁciency.
In this chapter we propose a new system called Kanji Tester which generates automatedvocabulary tests. Since tests are generated on demand, it meets the ideal test’s goal of availability. It sits at an intermediate level in terms of scope, since the types of questions it can
ask a learner are constrained by what can be reasonably automated. However, its intelligent error models allow it to generate a diﬀerent test every time, and thus to approach thescope of some human generated tests. Since the system allows a learner to test and re-test
themselves repeatedly, we call its tests drills.
To simplify Kanji Tester’s construction and to gain it a larger audience, we attempted toemulate the well-known Japanese Language Proﬁciency Test, which we now discuss.

6.2

Japanese Language Proﬁciency Test

Overview of the JLPTe Japanese Language Proﬁciency Test ( JLPT) is a family of examinations run biannually by the Japan Foundation. is section provides an overview; the interested reader
should visit the JLPT site1 run by the Japan Foundation (2009). It currently consists of four
separate exams at targeted at diﬀerent levels of diﬃculty, from Level 4 (easiest) to Level 1(hardest), as shown in Table 6.1. Level 1 is designed to be the level required to live and interact in Japanese society, and tests all the kanji in the government proscribed 常用 jyōyō “dailyuse” set which native speakers are expected to know upon completing secondary education.
e test itself consists entirely of multiple-choice questions. It contains three parts,
as shown in Table 6.2: Writing and Vocabulary; Listening; and Reading and Grammar.
From 2010, the writing section of the test will be merged in to the reading section which
will then contain 75% of the total points in the test. Multiple-choice tests such as the
JLPT give signiﬁcant scope for guessing of unknown answers . Since JLPT questions havefour potential answers for the learner to choose from, unknown questions may be correctly1

http://www.jlpt.jp/

Chapter 6: Testing and drilling
Level
4
3
2
1

# kanji # words
100
800
300
1500
1000
6000
2000
10000

133
Listening
Hours of study Pass markBeginner
150
60%
Basic
300
60%
Intermediate
600
60%
Advanced
90070%

Table 6.1: e four levels of the JLPT, with approximate numbers of kanji and words whichlearners should know when testing at each level, and estimated hours of study required.
From Japan Foundation (2009).
SectionWriting and vocabulary
Listening
Reading and grammar
Total

Points
100
(25%)
100
(25%)
200
(50%)
400 (100%)

Table 6.2: Sections of a JLPT exam and their relative weighting. From Japan Foundation(2009).
guessed 25% of the time. However, the standard correction for guessing is not applied
to ﬁnal test scores, but instead a higher pass mark is used to compensate. We note thatwith four options per question, a learner who knew the correct answer to half the questions
and guessed randomly the remainder should score 62.5% on average. 62.5% is thus thetheoretically appropriate break-even pass mark for such a test, rather than 50%. is ﬁgure
is approximated for the JLPT by a 60% pass threshold for levels 4, 3 and 2. For level 1, thepass mark of 70% more than suﬃciently corrects for guessing.
It is a criterion-referenced test. us, rather than measuring an individual against theirpeers (norm-referenced assessment) or against past performance (ipsative assessment), it assesses a learner against ﬁxed proﬁciency criteria (Begg 1997:22). is means that pass rates
for each level vary from year to year according to the proﬁciency of candidates, and also thatan outcome of the JLPT is a certiﬁcation of proﬁciency.
Several aspects of the JLPT make it suitable as a reference test for our purposes. JLPT
exams are based entirely on multiple-choice questions. Such questions can be easily andobjectively marked, although in principle could be constructed so as to be arbitrarily complex. In practice, the Writing and Vocabulary and Reading and Grammar sections bothcontain question types which plausibly be automatically generated. Together these sec-

134

Chapter 6: Testing and drilling

Figure 6.2: An example results card from the JLPT 2007 tests.
tions contribute 75% of the test, suggesting that with reasonable coverage of their question
types we should be able to estimate performance on the JLPT tests themselves, and throughthis measure proﬁciency. Although in this thesis we can only automate a limited numberof question types, we nonetheless see opportunity for extending this coverage in the future. Finally, the JLPT provides a cohesive audience for a Japanese testing system: roughly
560000 candidates undertook one of the JLPT levels in 2008, and for many of these candidates signiﬁcant study is required beforehand. By targeting these users we aimed to collectsigniﬁcantly more data than we would otherwise have been able to.

Question types
Each year, a new JLPT test is written for each of the four levels, and then administered
to a large number of candidates. Since the aspects of language proﬁciency being tested arelimited, and each question is multiple-choice, the number of question types used to test each
aspect of proﬁciency is ultimately limited. It is beyond the scope of this thesis to perform

Chapter 6: Testing and drilling135

a full account of these question types, but instead we focus on those which we believe best
test vocabulary knowledge directly. In our view, the most basic aspects of word knowledge
are the links between a word’s form (or surface), its meaning (or gloss) and its pronunciation(or reading), as given in Figure 6.3.

Words containing kanji

Words without kanji

Surface
Gloss
Gloss

Reading

Reading

Figure 6.3: Easily testable aspects of word knowledge for words containing kanji, and words
without kanji. A dotted line indicates a more tenuous link.
Many vocabulary-related questions in the JLPT interrogate these links. For example,
Figure 6.4 asks the learner to select the correct pronunciation for each word in the sentence,thus testing the surface-to-reading link for these words. e same link is tested in reverse
direction in Figure 6.5. Both questions make use of expert knowledge on behalf of test
authors, using their experience to choose distractors for each question based on plausible
misreadings ﬁrstly and plausible misrecognition secondly. Yet these are exactly the forms oferror models we have for Japanese, and which formed the basis of our improved dictionary
lookup in Chapter 5.
Since the JLPT is administered in Japanese, and is designed to accommodate participants from a variety of ﬁrst-language backgrounds, the links to L1 meaning are not tested
explicitly in the JLPT. However, since we choose a restricted audience for Kanji Tester –that of English speakers – we are also able to generate questions which interrogate the links
to meaning.
ere are many other question types which might plausibly be automatically generated.

136

Chapter 6: Testing and drilling

Figure 6.4: Example question: the learner must select the correct reading for each kanji
word in context. Taken from the JLPT level 3 example questions, Writing and Vocabulary
section.

Figure 6.5: Example question: the learner must select the correct kanji form for words givenby their reading. Taken from the JLPT level 3 example questions, Writing and Vocabulary
section.

Chapter 6: Testing and drilling137

For example, questions testing grammatical patterns might be generated according to manually created templates akin to Zock and Afantenos’s (2007) pattern drills, and similarly forquestions focused on correct and appropriate verb conjugation. Likewise, questions basedon reading comprehension might be constructed in the following way. Firstly, appropriate texts could be sourced online, for example news articles ﬁltered by diﬃculty using Satoet al.’s (2008) measure. Semantic entailment or automatic summarisation techniques could
then be used to generate sentences entailed by the text, to which the learner could give a trueor false answer. Likewise, variations using words from the text but which are not entailed
could be added as incorrect cases.
Despite the plausibility of these additional question types, they require signiﬁcant additional labour and expertise in order to automate their generation. Since our scope is limited,
we focus instead on the simpler vocabulary-based questions which might beneﬁt most fromthe error models used earlier in this thesis. e following section describes how the Kanji
Tester system presents and implements automatic test generation based on these simple
question types.

Potential criticisms of the JLPTWe have described an abstract conception of an ideal test for language learners, andfollowed on to describe the most frequently used family of proﬁciency tests for non-native
speakers of Japanese, the JLPT levels, which we use as a reference point for our test generation system. However, there remains a gap also between the JLPT levels and the ideal test
as we have characterised it. is section focuses on this gap, discussing potential criticismsof the JLPT and our use of it as a reference point.
e attributes of the ideal test include a broad scope, i.e. a goal of determining holistic
proﬁciency in a language. e JLPT does not provide a single test like this, but rather it
provides several tests, each of which assesses participants against criteria for a certain levelof proﬁciency. e choice of what levels to accredit and thus how many tests to run issomewhat arbitrary at all levels but Level 1, which is roughly pegged at the level of a native
speaker having completed Japanese secondary school education. ese arbitrary levels could
arguably be replaced by a single test, calibrated in such a way as to provide reliable proﬁciency

138

Chapter 6: Testing and drillingscores for a large variety of candidates of diﬀering ability.
e main arguments against such a large test are common to paper-based tests. Firstly,
it could be argued that a single test would have to be too long to accurately measure a wide
variety of diﬀerent ability levels, and would contain many questions which would either
be far too easy or far too hard for a learner. However, computer-based adaptive testing
would allow short tests for learners of a wide variety of levels. We note that the Test ofEnglish as a Foreign Language (TOEFL) is a single monolithic test, however it is adaptivein the listening and structure sections (Chalhoub–Deville and Deville 1999:287). Whilst
we expect all major proﬁciency tests to move to this approach in the future, we nonethelessnote that the JLPT is successfully in use as a proﬁciency test. e separation into four levelsof testing is useful for us, since it allows us to focus on the more limited levels 3 and 4
in Kanji Tester, rather than having to construct a full test system for assessing native-like
Japanese proﬁciency.
e JLPT can also be criticised for measuring only receptive aspects of proﬁciency, according to the receptive/productive distinction we introduced in Section 3.4. is largely
relates to the use of multiple-choice questions for vocabulary testing, since providing a missing word or pronunciation is harder than choosing from it amongst distractors. In particular, Laufer and Goldstein (2004) argue that this diﬃculty relates to the graded nature of
vocabulary knowledge; by not evaluating productive knowledge directly, the JLPT reduces
the depth at which it measures word knowledge. However, we noted in Section 3.4 thatbreadth and depth of knowledge are correlated, suggesting that objections based on depth of
testing may be overcome simply by testing a greater number of words. We further note thatmultiple-choice questions reduce marker bias, as opposed to productive questions which
require the learner to ﬁll in the blank. e use of multiple-choice only makes the JLPT
questions better candidates for automation in Kanji Tester.
More broadly though, the form of the test prevents the direct measurement of productive
oral ability. is means that, for example, an individual with excellent general vocabulary,grammatical and listening skills but extremely poor pronunciation might fare well on thetest, but poorly in the real-world interactions the test aims to mimic. is criticism is common to many proﬁciency tests, and the merits of including interactive oral testing remain up
for debate. Here, we merely hypothesise that for the average learner, their productive oralChapter 6: Testing and drilling139

skills may be well estimated by a combination of vocabulary, grammar and listening skills
which the test does cover. Even without such oral testing, the current cost of test-building
reduces availability of tests to learners. Introducing more costly oral testing is a trade-oﬀ
against other quality attributes of tests. In any case, the lack of interactive oral testing makes
the goal of automatic test generation more achievable in the medium term.
As this section has shown, most shortcomings of the JLPT are also beneﬁts for attempts
to automate JLPT-like test construction. e JLPT thus provides a meaningful yet strong
goal to aim for in test generation.

6.3

Kanji Tester

Having described in detail the JLPT examinations from which we draw our syllabi andquestion types, we now discuss our system for automatically generating tests for users, Kanji
Tester. We do this in two parts. Firstly we walk through the system from a user’s perspective,
describing the system as they use it. Secondly, we describe it from our perspective, looking
at its internal architecture and the mechanics of question generation and user modelling.

User interface
When a user accesses Kanji Tester for the ﬁrst time, they must sign up for an account.
is provides us with the means to customise tests to individual users. As part of the signup process, the user must create a proﬁle, as shown in Figure 6.6. is includes choosing asyllabus to study, which at the time of writing was either JLPT level 3 or 4. ey must also
give their ﬁrst language, and may optionally list any other languages they have studied. euser is then presented with a dashboard encouraging them to take a test. ey choose thelength of the test, from 10 questions to 50 questions, and click “Take a test”. Kanji Testerthen generates a new test for them based on their syllabus, as shown in Figure 6.7.
Each question in the generated test (a test item) is multiple choice, where the user mustselect the single correct answer amongst distractors, and is based on a single word or kanji
from the syllabus. Although there are many aspects of word knowledge, Section 3.1 discussed Vermeer’s (2001) work linking breadth and depth, indicating that well constructed

140

Chapter 6: Testing and drilling

Figure 6.6: Creating a proﬁle for Kanji Tester. e user enters a ﬁrst language, and optionally any second languages they have studied.

Chapter 6: Testing and drilling

Figure 6.7: A sample test generated by Kanji Tester for JLPT 3.

141

142

Chapter 6: Testing and drilling

breadth testing was suﬃcient to measure proﬁciency. For this reason, only a few basic but
crucial aspects of word knowledge are tested.
Figure 6.3 shows the types of links available between the three forms of word knowledge,
each of which represents an aspect of knowledge which can be tested, and each of which
generates a diﬀerent question type depending on the direction the link is traversed. For
example, surface-to-reading yields a question where the user is presented the surface as
stimulus, and asked to identify the correct reading (Figure 6.8); gloss-to-surface would use
the gloss as stimulus, and ask the user to identify the correct surface. In the case of words
without kanji, the reading and the surface components are the same, since kana are syllabic;
this reduces the available links to only the gloss-reading link.

Figure 6.8: An example question measuring the surface-to-reading link for the word
用意 yōi “preparation”.
e user selects their answers to each question in the test and then clicks “Check answers”. ey are then provided with feedback on their test performance, as shown in Figure 6.9, which lists their score and highlights mistakes. If the user clicks on “Study mistakes”, they are provided with a vocabulary list of any word or kanji they made a mistakeon, complete with multiple readings and glosses. is study list is tied to the user’s syllabus
in two ways, ﬁrstly in the sense that it can only contain words and kanji which the user wastested on and are hence in their syllabus, and secondly in the sense that only aspects of word
and kanji knowledge within the chosen syllabus are shown in the study view. In particular,a word with multiple pronunciations and kanji forms may have some of these excluded fromthe study view (and from testing) if they are not part of the syllabus. Ideally, word senses
would be similarly restricted, however the JLPT syllabi do not specify words or kanji to thislevel.
Once the user returns to the dashboard, they are presented with two main statistics
on their performance: their cumulative accuracy, and their accuracy on the most recentChapter 6: Testing and drilling

Figure 6.9: e feedback given to a user immediately upon completing a test.

143

144

Chapter 6: Testing and drilling

test. Ideally, if their performance is improving through study, their most recent test score
should be higher than their cumulative average, increasing motivation for the user. eir
accuracy on these tests is designed to itself be a proﬁciency estimate of receptive vocabularyknowledge. Cumulative accuracy is thus the best estimate of the user’s average proﬁciency
over the entire period they have used the system, true to their ability but slow to respond tochanges. e last test taken provides a dynamic estimate which responds quickly to changes
in proﬁciency but which is far noisier.
Note that in our heavy use of multiple-choice questions, we do not allow users to omit
answers and neither do we use the standard correction for guessing in the user results, whichtakes the form of a penalty for incorrect answers. is approach was taken for several reasons. Firstly we note that no such correction is taken on the JLPT, which we emulate,although participants may omit answers. Secondly, the standard correction formula2 assumes that in cases where the learner doesn’t know the answer, the learner guesses randomly
amongst the available options. is assumption is clearly false in cases of partial knowledge,
and the modelling is complex to determine the appropriate penalty for such cases (Espinosaand Gardeazabal 2007). irdly, using a penalty and allowing users to omit answers would
reduce guessing, and whilst this would decrease the number of explicit errors (as opposed
to omitted answers), these errors are necessary to feed our adaptive error models. Finally,such corrections are typically aimed at determining the participant’s ability in terms of their“true score” on a given test; we instead use far more ﬁne-grained user modelling to assess ausers’s proﬁciency, making correction unnecessary from an ability estimation perspective.

User modelling
Kanji Tester shares many aspects of Intelligent Language Tutoring Systems, such as
Heift and Nicholson’s (2001) German Tutor, with the main diﬀerence residing in Kanji
Tester’s restricted focus on vocabulary rather than grammar. A large similarity though resides in the reliance of such systems on an accurate user model to base user interaction on.
A range of user characteristics can be modelled, including goals, knowledge, background,2

e standard correction for guessing estimates a user’s corrected score s on a test with k options as s =w, where r is the number of correct responses and w is the number of incorrect responses. is correction
r− k−1
1
is eﬀectively a penalty of k−1
for each incorrect answer.

Chapter 6: Testing and drilling145

preferences, interests, individual traits and environment (Brusilovsky 2001). Kanji Tester
has two of these key elements available to it, namely the user’s ﬁrst and second language
background, and their vocabulary knowledge, as displayed in their test performance.
Each question a user answers is based on a particular item in the syllabus, a word ora kanji, and requires distractors to be generated as plausible alternatives to the the correct
answer. e plausibility of the distractors is crucial; if users can systematically guess thecorrect answer without knowledge of the item in question, the test will have no validity andwill not be able to measure proﬁciency. Choosing an appropriate user model is thus crucial
to generating useful distractors.
In order to construct distractors for each question with similar credibility to what ateacher would select, we make heavy use of both the plausible misreading and the plausible
misrecognition employed in our rebuilt version of FOKS. However, with a continuous inﬂux
of user responses we have the potential to improve these models, making them respond andadapt to user mistakes. In particular, the system can increase its diﬃculty by increasing thefrequency of distractors or error types which users have made mistakes on. To do this weneed to perform at least limited modelling of user knowledge and behaviour.
ere are several layers of granularity at which such user modelling could potentiallyoccur, both in terms of errors and in terms of users, as shown in Figure 6.10. Users could
be modelled: (a) globally, thus adapting to the user population as a whole; (b) in groups, forexample by ﬁrst language background; or (c) individually, adapting to each user. e larger
the user grouping, the greater the beneﬁt to users who are representative of that group, andthe greater the penalty to minorities and individuals with error proﬁles which diﬀer fromtheir group. If users are considered individually, data sparseness becomes an issue, and thelevel of adaptivity may be limited.
A good example is provided by vowel length errors. Since English has no vowel length
distinction, native speakers of English may be less sensitive to vowel length in Japanese,and thus make more errors. A global shared model would adapt to the background of the
majority of the learner base, in our case English speakers, thus penalising learners from otherbackgrounds who might not make these errors. If our model is based instead on language
background, then other language groups are not penalised, but individual native English
speakers who are attuned to vowel length diﬀerences still are. If we model errors for each

146

Chapter 6: Testing and drilling

User granularity

Error granularity
correct

globalincorrect

Spanishspeakers
subgroups

Englishspeakers

Chinesespeakers

Choice ofreadingSoundeuphony

Palatalization

Vowel
lengthkanji level
tō

to

東

東

✔

✘

kyo

kyō

京

京

✔

✘

word level
individuals

tōkyō

tokyo

東京

東京

✔

✘

Figure 6.10: Examples of user and error models of varying granularity considered for use
by Kanji Tester. On the left, user models can be maintained globally (treating all users
identically), in aggregate by some common attribute (e.g. ﬁrst language), or individually.
Similarly error prevalence can be modelled globally, by broad error type, by kanji or by word.
e actual conﬁguration of Kanji Tester as reported models users as individuals and errors
at the kanji level.

Chapter 6: Testing and drilling147

user individually, no user is penalised by the behaviour another, but they cannot beneﬁt fromother users either. For Kanji Tester, we chose this last form of error model, as advocated by
Rich (1983), because out of our three main options it makes the least assumptions about
each individual user’s behaviour.
For any level of user model, there are four possible granularities of error models which
Kanji Tester could potentially use. Suppose the user chooses the incorrect reading tokyo
for the word 東京 tōkyō “Tokyo”. e most coarse model simply registers this as an error,
without caring about the error type or the word involved, and notes that the learner is now
more likely to be wrong again in the future. A slightly ﬁner grained model would register
the error as one of incorrect vowel length, but would still ignore the details of the word
involved. It would note that the learner is more likely to make vowel length errors in the
future. A more detailed model at the kanji level would register that the learner had in fact
made two errors, misreading 東 tō as to and 京 kyō as kyo, and would estimate the likelihoodof the learner misreading either kanji this way as increased. e most ﬁne-grained model
would consider the error as a speciﬁc to the word 東京, and increase the likelihood of thismisreading for this word only.
e main question here is the extent to which error occurrences can be generalised into
useful broader trends. In Kanji Tester, we model error trends at the kanji level and use theseper-kanji models in sequence when kanji occur in sequence as words. is is convenient
since error models from FOKS are applied at the per-kanji level, and thus transfer directly.
Error models from FOKS take the form of probability distributions. For a word containing kanji, with r as its guessed reading, k as its kanji form, and k ′ as a plausible misrecognition, the models specify Pr(r|k) and Pr(k ′ |k). For the misrecognition model, we used thesame error model we used for FOKS, incorporating stroke edit distance as the similarity
measure. In Kanji Tester, we simply use a static snapshot of these models as prior distributions, and maintain for each user u their own dynamic posterior distributions Pru (r|k, R)
and Pru (k ′ |k, R), where R is the ordered sequence of their previous responses. Note that for
simplicity this snapshot removes any distinctions between error types, and simply considers
them as error candidates with diﬀerent likelihoods. is is aimed at providing a reasonable
trade-oﬀ between data sparseness and adaptability for each individual user. us, in theearlier case of 東 tō being misread as to, our kanji-level reading model would only note it as

148

Chapter 6: Testing and drilling

an incorrect reading rather than as a vowel length error. e user model would then update
itself so as to register the likelihood of 東 being misread this way in the future, rather than,say, modelling vowel length errors in general as increasing in likelihood for this user.
e following two sections describe in more detail ﬁrstly how our user models are used
to generate questions, and secondly our update algorithm for reﬁning our error models fromuser responses.

Generating questions
In order to better describe how Kanji Tester generates test questions, a brief overview
of its architecture is useful. Kanji Tester provides an online web-based user interface fromwhich users can choose a syllabus and take tests of 10, 20 or 50 questions in length. Each test
is generated on a question-by-question basis, using three key components of Kanji Tester:
its learner syllabi, its user models and its question plugins. ese correspond to the dottedboxes in Figure 6.11’s architectural overview, and are designed to be extensible. Each ofthese three components has a crucial role in question generation.
Each syllabus supported by Kanji Tester is prepared for use in the form of a paired word
and kanji list. Each word on the word list is matched to a word from a reference lexicon– in our case JMdict – in a semi-automatic fashion. Once a user has chosen a syllabus to
study, every question generated for them is seeded by a word or kanji from that syllabus’s list.
is is the ﬁrst step in Kanji Tester’s question generation algorithm Figure 6.12. e next
choice in question construction is what type of question to generate. Individual questions
are generated by question plugins, and for each test only a single plugin can generate eachquestion type.
Tests silently alternate between two types, control and adaptive. is mirrors the two
types of question plugins available, simple and adaptive. Simple plugins represent a baseline
eﬀort; their question generation methods are fundamentally similar to adaptive plugins, but
are static and unchanging. On the other hand, adaptive plugins make use of user models
which they reﬁne with each user response. Control tests are limited to using only simple
plugins, whereas adaptive tests make use of adaptive plugins wherever possible.3 e con3

e main exception to this the “Random gloss” plugin, which is a simple question plugin used in both

Chapter 6: Testing and drilling

149

User interface

Drill generator

QUESTION PLUGINS
Random gloss
SIMPLE

Random reading

Reading alternations
ADAPTIVESIMPLE

Random surface

Visual similarity

SIMPLE

USER MODELPLUGINS

ADAPTIVE

Lexiconkanji' | kanji

SYLLABIreading | kanji
JLPT 3JLPT 4

Figure 6.11: e high-level architecture of Kanji Tester. Dotted lines indicate areas designed for easy extension.

150

Chapter 6: Testing and drilling

1. Randomly select a seed item from the user’s syllabus.
2. Randomly select a question plugin, of the available plugins which
are capable of using the given seed item.
3. Within the question plugin:
3.1. Randomly select a question type from those the plugin can
generate.
3.2. Generate a pool of valid distractors, optionally using one of
the available user models.
3.3a. [Simple plugins] Randomly sample distractors from thepool of available distractors with uniform probability.
3.3b. [Adaptive plugins] Randomly sample distractors fromthe pool of available distractors with proportional probability.
4. Render the resulting question to the user.
5. [Adaptive plugins] Update user models with the user’s response.
Figure 6.12: e algorithm used by Kanji Tester to generate each test question.

Chapter 6: Testing and drilling151

trol/adaptive distinction is designed to allow the comparison of the two methods, with the
expectation that adaptive tests should be more diﬃcult than their control counterparts.
Regardless of whether the test is a control test or an adaptive one, a ﬁxed set of questionplugins is then available to generate questions with. If the seed item is a word containing
kanji, any plugin from that set can be used. If the item instead uses no kanji, then only thesimple gloss plugin can be used. If several plugins are available, one is chosen randomly
from amongst them. is corresponds to step 2 in Figure 6.12. If the chosen plugin can
generate more than one type of question, it then chooses randomly between them, and forthe ﬁnal question type it generates a pool of distractors. From this pool, ﬁve are randomly
chosen to be presented alongside the correct answer. In simple plugins, distractors are chosen with uniform probability; in control plugins, they are chosen with probability equal totheir estimated likelihood of eliciting an error. e question is then presented to the user,
and their response recorded. If the plugin was adaptive, the user’s response is ﬁnally fed
back into the user model, which updates itself according to the update rule we discuss.

Update algorithm
e basic goal of our approach to testing is to attempt to maximise the number of genuine errors we provoke from the user. By making them aware of gaps in their knowledge,
we provide them with the opportunity to correct these gaps. We are aware that diﬃcult tests
can reduce study eﬀort (Marso 1969), an eﬀect reinforced by more recent work by Ponsoda
et al. (1999) on manipulating the diﬃculty of computer-adaptive tests, and this suggeststhat we may wish the tests to be easier for learners. However, multiple choice test diﬃculty
can always be reduced by using less plausible distractors. On the other hand, ﬁnding moreplausible distractors to increase test diﬃculty for the same target words is far more cumbersome. We take the reasonable approach of simply maximising diﬃculty (and hence user
errors), knowing that even with good distractors our tests may err on the easy side.
In order to do this, we model each user’s knowledge and discriminative ability using theinformation we have available: our prior expectation of what common errors will be made,and the user’s actual responses to previous questions. is section is concerned with making
control and adaptive tests, since no adaptive counterpart exists.

152

Chapter 6: Testing and drillinguse of the latter information.
Our update algorithm takes an error model, the question it generated and the user’s
response to that question, and uses these elements to construct a new updated error modelthat better ﬁts the learner’s observed behaviour. More intuitively, it should ensure that
distractors which successfully confused a user are presented more frequently than distractors
which did not. ere is a large space of algorithms which could perform this function. emain trade-oﬀ any such algorithm faces has to do with the rate of adaptivity. If the algorithm
adapts swiftly to user input, it is also more likely to over-ﬁt noise; if it adapts too slowly, thestrength of the model will rely entirely on its priors rather than on per-user information.
Our user-level granularity tells us what error model to update with each user response,and our error-level granularity gives us a concrete probability distribution. We now describe
the update algorithm employed in Kanji Tester, which compares each user response to theprobability distribution(s) which generated it and updates the distribution(s) so that the
response is considered to be more likely to occur in the future.
We start with basic deﬁnitions:w = the word the question is based on, w = k1 . . . kn
O = the distractor space for the word, O = O1 × · · · × On
D = the options shown to the user, D ⊂ O
c = the user’s chosen answer, c ∈ D
We displayed a random subset D of possible options O to the user, and they chose optionc as their answer. According to our user model, each d ∈ D had its probability calculated
ﬁrstly by:Pr(d|w) = Pr(d1 . . . dn |k1 . . . kn )n∏≈Pr(di |ki )

(6.1)

i=1

and then normalised ∀d ∈ D to get Pr(d|D). In this way, we already know the value
Pr(c|D). We now wish to determine the posterior distribution Pr ′ for (C|D). We base our
update rule on the constraint:∀{d:d∈D\{c}} Pr ′ (c|D) ≥ Pr ′ (d|D) + ϵ(6.2)

Chapter 6: Testing and drilling153

at is, the user chose c because it was better than any other option by a margin of ϵ. If weassume that the user did not guess at random, but chose their answer because it was the most
likely one to be correct out of the options available, then this rule simply ensures that in theposterior distribution their answer becomes the most likely out of the given options. It does
this by enforcing a margin of ϵ in the posterior distribution (C|D) used in the next iteration.
Note that ϵ serves to smooth the update of the error model, and the value chosen represents
a particular tradeoﬀ between quickly adapting even to noise and random guesses, or slowly
adjusting to learner input. We used an ϵ of 0.2, which we intended as an intermediate value.
e use of an ϵ margin was also intended to indicate that not every question we ask
the user yields new information. If the ϵ margin already exists between the user’s choice
and the other options, then the current model adequately predicted the user’s response, andno change is needed. If the margin does not exist, we update the error model using thefollowing steps:
1. Let m = max{i:di ̸=c} Pr(di |D) + ϵ
2. Deﬁne the posterior distribution (C|D) as follows:• Pr′ (di |D) = Pr(di |D) if di ̸= c
• Pr′ (c|D) = max{Pr(c|D), m}∑• Normalise (C|D) such that i Pr′ (di |D) = 1
Now we know the posterior distribution of (C|D), yet our error model is stored interms of (Ci |Di ). In other words, ∀d ∈ D, we know the sequence probability Pr ′ (d|D) =Pr ′ (d1 . . . dn |D), but still need to deﬁne Pr ′ (di |Di ). at is, we need to distribute thediﬀerence between the prior and posterior distribution for (C|D) to each (Ci |Di ). We do
this by ﬁrstly deﬁning the constant ∆:∆ =≈Pr ′ (d|D)
Pr(d|D)n∏
Pr ′ (di |Di )i=1

Pr(di |Di )

(6.3)

154

Chapter 6: Testing and drilling

We reach Equation 6.3 above by replacing Pr ′ (d|D) and Pr(d|D) with their sequence
approximations from Equation 6.1. We are now close to being able to redistribute thisprobability mass. We add the additional approximation that the mass should be distributed
evenly amongst sequence components:
Pr ′ (dj |Dj )Pr ′ (di |Di )=Pr(di |Di )Pr(dj |Dj )∀i, j ∈ 1 . . . n(6.4)

Combining this with Equation 6.3, our ﬁnal update rule emerges:
Pr ′ (di |Di ) = ∆ n Pr(di |Di )
1

∀i

(6.5)

is update rule has some desirable properties. For example, only distractor components
displayed to the user have their likelihood changed. is is true at the kanji level, though thelikelihood of unseen kanji sequences may still change. Suppose the user had to identify theword meaning “composition, writing”. e correct answer is 作文, which they must identify
amongst distractors generated by our (kanji′ |kanji) misrecognition model. ey make an error and choose 非人 as their answer. In response to their answer, probability mass is moved
away from the other word-level (mis)recognition candidates displayed – including the correct answer – and distributed to (非人|作文). In turn, this additional probability mass is
distributed by our update rule into increased likelihoods for pairs (非|作) and (人|文). isalgorithm thus provides a principled method for updating user error distributions at the
kanji level based on their answers to test questions.
In order to distribute probability mass to reading parts, each word in the syllabus containing kanji must be accurately GP-aligned to its reading. We used the unsupervised
method described in Section 5.2 to automatically align each syllabus, and then manuallycorrected any alignment errors encountered. e only exception to our update rule lies inour (reading|kanji) user model, in the case of non-compositional readings. If the user must
guess the reading of a word where the correct answer is non-compositional, as in the case of
山車 dashi “festival ﬂoat”, a correct answer from the user will have have no way of attributing the reading of the whole to the reading of its kanji parts, and thus no way to increase
the likelihood of the correct reading in place of an incorrect reading. In this rare and specialcase, no update takes place. However, this is not problematic since the correct reading must

Chapter 6: Testing and drilling155

always be amongst the available answers to a question.

6.4

Evaluation

In Section 6.1 we discussed our goal of replicating human constructed tests, and thendiscussed our method of using the error models from FOKS to generate plausible distractors
for diﬀerent question types. After deploying the Kanji Tester system in November 2008,
roughly one month before the 2008 JLPT tests, we advertised it on several mailing lists
and used a small Google AdWords campaign to try to gain users who would be studying
for the test. Since Google trends indicated that the search term “JLPT” encountered heavy
traﬃc in Singapore, we focused our limited AdWords budget on Singaporean users in an
attempt to gain participants in a short timeframe. After the JLPT season was over, weanalysed log data collected between November 2008 and February 2009 in an attempt tobetter understand this new system, how it was used, and the extent to which it helped users.
is section discusses the details of this analysis.

User demographics
During the period under analysis, 225 users completed a minimum of one test, responding to 17065 questions in total (75.8 questions per user on average). In order to use thesystem, each user ﬁrst had to enter their ﬁrst and second language background. Table 6.3
shows that, although most users had English as their ﬁrst language, we had users from 36
other language backgrounds, together accounting for 60% of the user population. Notethat this includes 19 people of Japanese ﬁrst language background, who may have been
language educators experimenting with the system. For the remainder of this chapter we
exclude these users from our analysis, since their responses are unlikely to be characteristicof typical learners.
Of the two syllabi available to users, 71% of users chose the more advanced JLPT level3, whereas 29% of users chose JLPT level 4. As with any web-based system, participation
rates were not even between users. Figure 6.13 gives the number of users that have taken atleast n test, for increasing n. e majority of users completed less than ten tests in total. AChapter 6: Testing and drilling
Language# Users
English90 41.3%
German28 11.9%
Japanese
19 8.7%
Marathi
12 5.5%Indonesian9
4.1%
Vietnamese
6
2.8%
Chinese6
2.8%
Other (30 more) 50 22.9%
Total
219 100%
Table 6.3: Distribution of active users by ﬁrst language.

small handful completed many more.
# users
User volume vs number of tests
225
250
141
88
200
70
58
48
150
43
39
100
34
32
50
26
23
21
0
0
10
20
30
40
50
60
70
19
18
# tests
16
14
Figure 6.13: Number of users (reverse cumulative) set against number of tests.
13
12
11
10 Also of interest is how these users made use of the system. Did they test themselves
9
repeatedly over a short space of time, or return to the site occasionally over longer periods to7
determine their progress? To measure this, we ordered the tests each user took sequentially6by their timestamp, and measured the time between sequential tests. Figure 6.14 shows a5
4
reverse cumulative histogram of the mean time between tests for each user, with time on a3
2
1

# users

1
2
3
4
5
6
7
8
9
10
11
13
14
15
16
17
18
19
20
22
24
27
28
32
33
34
40
45
67

156

Chapter 6: Testing and drilling

Proportion
1.00
0.99
0.78
0.48
0.36
0.28
0.24
0.21
0.18
0.18
0.16
0.14
0.10
0.06
0.03
0.02
0.01
0.00

Mean time between tests (reverse cumulative)
1.00

0.75
Proportion

Days
.00
.00
.00
.00
.01
.01
.02
.04
.09
.18
.36
.71
.42
.84
.69
1.38
2.76
5.51

157

0.50

0.25

0

0.01

1.00

100.0010,000.00

Hours

Figure 6.14: e proportion of times between tests taken sequentially by a user, expressedas the proportion of such times of magnitude > x hours (reverse cumulative histogram).
Hours
0.00
0.02
0.03
0.07
0.13
0.27
0.53
1.07
2.13
4.27
8.53
17.07
34.13
68.27
136.53273.07546.13
1,092.27

Proportion
log scale.
1.00 It shows that the vast majority of tests are taken in quick succession to the previous
test,0.99
within a few minutes. is suggests that many users used the system as a drill rather
0.78
than0.48
a proﬁciency test, and thus that we will be unable to measure long-term progress for
0.36
most users due to the short time frames.
0.28
0.24
0.21
0.18
0.18
Kanji
0.16 Tester is designed to help learners self-evaluate their proﬁciency; for this reason,
0.14
items
for tests were chosen randomly from the user’s chosen syllabus, as discussed in Sec0.10
tion0.06
6.1. Over the course of many such tests, Kanji Tester gains an increasingly accurate
0.03
picture of a user’s knowledge level. If we take our best estimate of their ability, i.e. their av0.02
erage0.01
accuracy across all responses, we get scores as shown in the histogram in Figure 6.15.
0.00

User ability and improvement

e majority of users in both tests had mean accuracy greater than 0.75 over all tests taken.

is ﬁgure is representative of their likely performance on the equivalent portion of the
JLPT, namely multiple choice questions of this form. However, the JLPT is designed totest various facets of Japanese knowledge, so it is unlikely that we could accurately predict

158

0
4
9
17
6
28
82

0.35
0.45
0.55
0.65
0.75
0.85
0.950000005

0
0
5
6
1
14
39

Chapter 6: Testing and drilling

performance on the whole test from just receptive vocabulary knowledge.

User proficiency histogram90

Number of users

0.35
0.45
0.55
0.65
0.75
0.85
0.950000005

67.5

45

22.5

0

0

0.25

0.5

0.75

1

Syllabus coverage

JLPT 3JLPT 4

Figure 6.15: Mean score per user as a histogram.
We are of course interested in how this proﬁciency is related to usage of Kanji Tester,since the premise of our work is that better self-evaluation will allow users to make better
decisions about their study patterns, ultimately improving their proﬁciency. If we compare
our best estimate of ability against the number of tests each user takes, we get the results
shown in Figure 6.16. Here, we see again that most users take few tests, and already scorequite well. A linear trend line shows a very weak positive correlation between usage andproﬁciency estimates.
Beyond static estimates of user ability, user tests are also distributed in time, and thus
can serve to measure change in ability over time. We now examine changes in ability inseveral ways. Firstly, we can use the number of tests taken as an estimate of study time, andcompare study time and ability. Figure 6.17 takes this approach, and shows a very weaknegative correlation between ability and study time as measured.
Although it is theoretically possible that increased study with poor study habits could
reduce ability, a more likely interpretation of this graph is that one or both of our measures

Chapter 6: Testing and drilling

159

Mean score vs number of tests
1

Mean score

0.8
0.6
0.4
0.2
0

y = 0.0019x + 0.8267
R! = 0.0069
0

10

20

30

40

50

60

70

Number of tests taken

0.02
0.1
0.2
0.3
0.3
0.3
0.3
0.3
0.3
0.4
0.4
0.4
0.4
0.48
0.5
0.5
0.5
0.5
0.5
0.5
0.6
0.6
0.6
0.6
0.6
0.6
0.6
0.6
0.6
0.6
0.6
0.6

Figure 6.16: Ability, as measured by mean score for each user, plotted against usage, as
measured by the total number of tests each user took.

Score on nth test
1
0.8

Score

Score

0.6
0.4
0.2
0

y = -0.0015x + 0.8743
R! = 0.0123
0

10

20

30

40

50

60

70

Number of tests

Figure 6.17: Ability, as measured by test score on the nth test, plotted against study time,
as measured by number of tests taken n, for all users.

160

Chapter 6: Testing and drillinglack full validity. For example, we saw in Figure 6.14 that most tests are taken in rapid
succession. If users lose interest or get tired (user fatigue), using test scores to measureability could lose validity, since we may be partially measuring user attentiveness instead.
Similarly, if tests are taken in rapid succession in a single session, the number of tests taken
may not be a good measure of broader time spent studying.

Mean score

Score
Scores over time
0.6666666667
1
0.725
0.9384615385
0.8
0.9333333333
0.775
0.9583333333
0.6
0.5666666667
0.9333333333
0.85
0.4
0.7
0.9769230769
0.2
0.85998
y = 0.0004x + 0.8648
0.8846153846
R! = 0.004
0.9333333333
0
0
20
40
60
80
0.95
0.88
Time after first test (days)
0.8
0.9
Figure 6.18: User scores plotted against time in days since their ﬁrst test.
0.75
0.85
0.9636363636
To address these concerns about measurement validity, we now plot user scores on tests
0.7
0.8 against real time in days since their ﬁrst test (Figure 6.18). Each point represents a user’s
1.0
mean score on a single day in which they took tests. e linear trend-line shows a very weak
0.5
0.84 positive correlation between ability and time passed, indicating that users are in general
0.9 improving in proﬁciency over time.
0.7666666667
Compared to ﬂashcard-style progression through a syllabus where users are continu0.9091
0.825 ously shown the same items until they are reliably learned, users of Kanji Tester encounter
0.8461461538
new items constantly, since items are chosen randomly from the syllabus. However, with
0.9285714286
0.86 suﬃcient tests, many items are encountered and tested several times. One way to determine
0.9333333333 if, general improvement aside, users are improving on the actual items they are tested on is
0.82
0.95
0.9
0.8
0.9333333333
0.7666666667
0.9
0.9636454545
0.9714285714

161

to compare their responses to such items over time. Since data is quite sparse, we looked atonly users who had encountered at least one item multiple times, and compared the meanscore on their ﬁrst encounter to such items to the mean score on the last encounter. e former should measure the knowledge they had before using the system; the latter represents
any change.

Pre-score post-score difference# users
0
1
0
3
4
32
11
8
5
0
3

40

Number of users

Pre-post
ifference.031818187.025454545.019090909.012727273.006363636
0.00063636364
0127272727
0190909091
0254545455
0318181868

Chapter 6: Testing and drilling

30

20

10

0
-0.04 -0.03 -0.02 -0.01

00.01 0.02 0.03 0.04

Difference

Figure 6.19: e diﬀerence in scores between the ﬁrst time a stimulus was seen and the lasttime it was seen, expressed as a histogram.
e results of this analysis are shown in Figure 6.19. ey show that on average there
is a slight improvement in accuracy on stimulus tested more than once, although for manyusers there is no diﬀerence. is in turn suggests either that few users use the provided“study view” and carefully study their mistakes, or that the majority of stimuli encountered
twice were those which users already knew.
To eliminate these potential eﬀects, we modelled learner knowledge as a ﬁnite state machine, where each word or kanji is either known or unknown for a given user when tested.
A correct answer upon subsequent encounter with the stimulus indicates a transition to
the known state, an incorrect answer a transition to unknown. We populated the modelwith response data by examining again all stimulus pairs that an individual user encountered at least twice, and ordered that user’s responses temporally into a sequence such as

162

Chapter 6: Testing and drilling

0.06
0.94

Known

Unknown

0.25

0.75

Figure 6.20: State transition diagram for known and unknown items. In this context“known” mean the user answered the most recent question about the item correctly.
⟨incorrect, correct, correct⟩. We used the ﬁrst response in each sequence to determine thestarting state, and each subsequent response as a transition. Assuming that each stimulus
item had an equal chance of being learned (or not) after each test, we combined data fromall stimulus items into a single state transition model, given in Figure 6.20.
e results are encouraging: known items have only a 6% chance of becoming unknown,
but unknown questions have a 75% chance of becoming known. However, this model has
limits on its predictive capacity. Exploring its performance characteristics, we ﬁnd that the0.75= 0.93, even after it has beenlong-run likelihood of getting any item correct is 0.06+0.75

tested many times. A previously unknown item reaches this level after it has been tested
just three times. is artefact, a residual 7% error rate, suggests limits to the model. In
particular, changes in user response do not always indicate changes in user knowledge, due
to two main eﬀects. Firstly, some correct answers are actually correct guesses. After a correct
guess, the learner will not be prompted to study the item, and is thus less likely to improvetheir knowledge for subsequent tests. Secondly, knowledge of a kanji or word is graded andmulti-faceted. In reality, a learner may easily recall one aspect of word knowledge (e.g. its
meaning) but not another (e.g. its pronunciation). Success or failure on questions basedaround the same word may purely reﬂect the diﬀerent aspects of word knowledge being
tested, rather than indicating changes in word knowledge.
Ideally, both of these problems could be overcome through the addition of multiple
states to our model. In the ﬁrst case, states could be added for each of main aspect ofknowledge, in particular for the links we showed earlier in Figure 6.3. In the second case,states could be added indicating two, three or four successive correct answers, in order toChapter 6: Testing and drilling
User #
305.
251.
191.
167.
253.
AverageFirst
language
EnglishEnglishShonaGermanGerman[English]

163

JLPT

# responses# tests

L3
L4
L3
L3
L4
[L3]

1160
860
670
450
440
75.7

24
18
67
45
16
4.99

Mean
accuracy
0.839
0.914
0.666
0.911
0.925
0.880

Pre/post
diﬀerence
0.056
0.056
0.087
-0.058
0.033
0.110

Time
period5 days12 hours
66 days62 days
27 days
3.7 days

Table 6.4: Basic user statistics for the 5 users with the most responses.
capture both the graded nature of knowledge and the weaker relationship between a correct
answer and item knowledge due to guessing. Unfortunately, since Kanji Tester tests words
and kanji randomly from the syllabus, data sparsity issues prevent us from examining thesericher models.

Power users
Having examined estimation of user ability across the entire user population, we now
restrict our focus to so-called “power users”, the top 5 users in terms of number of tests
taken. We provide some basic descriptive statistics for these users in Table 6.4, comparing
them to the general user population.
ese users diﬀer in nearly every aspect, giving little indication of what motivated theseusers to use the system more than others. eir ﬁrst languages include English and German,
the largest two language groups, but also Shona, a Bantu language. ey also include bothsyllabi supported by the system, JLPT levels 3 and 4.
Each user performed 6-15 times more tests than the population average, but varied
widely as to the length of test they preferred: User 305 used almost exclusively tests with
50 questions, and User 191 used tests of 10 questions. Our power users vary around themean accuracy level, with User 191 in particular having a very low accuracy compared tothe population average. All of these users continued for long enough to be tested randomlyon the same item twice, and thus can be given a pre/post diﬀerence score averaged across
such items. User 167 was the only user amongst the ﬁve to show negative improvement;
the other four all registered improvement, although less than the population average. is164

Chapter 6: Testing and drilling

would seem counter-intuitive: if increased use is beneﬁcial to users, then we would expect agreater-than-average improvement in the ability of power users. However, many users in thegeneral user pool never encountered an item more than once, and those that did typicallyencountered few such items. e smaller this number of items, the noisier the pre-postestimate is for that user is. is suggests that the population average on this measure may
be skewed in magnitude due to data sparsity.
Figure 6.21 gives an alternative view of each of the ﬁve power-users, plotting both thetotal number of unique stimulus items each was exposed to at the nth test, and the number
they answered correctly on the last encounter. e graphs should be interpreted with anunderstanding of two caveats. Firstly, since tests taken diﬀered in size according to eachuser’s preferences, the number of items actually tested per data point can vary from one to
the next. is could manifest itself in diﬀering slopes at diﬀerent points in time. Secondly,
this eﬀect could be compounded since the number of tests taken is only a rough indicationof the passage of time: the time between two neighbouring tests could be a matter of minutes or weeks. With these provisos, we note that the “Tested” lines display the asymptotic
behaviour we would expect from randomly choosing stimulus: they gradually approach thetotal number of kanji or words in each syllabus, but with each test the likelihood of reencountering a previously seen item increases, hence the decreasing slope as the asymptote
is approached.
e ratio between the number tested and the number correct when last tested is alsointeresting to observe between users. At any point in time, the ratio can be considered an
estimate of the user’s “true knowledge” of the syllabus’s vocabulary at that point in time. For
some users, for example user 251, a gap develops between the two lines for kanji knowledge,
indicating a number of errors made. e gap later closes when those kanji are retested onlater encounters. For user 191, a large gap emerges between the two lines and widens withincreasing coverage of the syllabus, indicating the lower proﬁciency reached at the pointof ﬁnal testing. Interestingly, the ﬁnal ratio of each user’s lines visually agrees well themean accuracy measurements for each user in Table 6.4, particularly on their word graphs:
user 191 performs most poorly, followed by user 305. e remaining three power users arecomparably high. User 191, who according to our pre/post ratio estimates improved themost over the testing period, also had the most room to improve. In comparison, the otherChapter 6: Testing and drilling

165
Kanji

User 305

300

700

225

525

150

350

75

175

0

User 251

0

15

0

20

60

375

40

250

20

125
0

5

10

0

15

200

500

150

375

100

250

50

125
0 5 10 15 20 25 30 35 40 45 50 55 60 65

0

200

400

150

300

100

200

50

100

0

User 253

10

500

0

User 167

580

0User 191

Words

0

5

10 15 20 25 30 35 40 45

0

70

300

52.5

225

35

150

17.5

75

0

0

5

10

Tested

15

0

0

5

0

10

5

15

10

20

15

0 5 10 15 20 25 30 35 40 45 50 55 60 65

0

0

5

10 15 20 25 30 35 40 45

5

10

15

Correct

Figure 6.21: Accuracy and exposure for words and kanji by the top-5 power users. Eachgraph shows the total number of unique items tested after the nth test, and the number of
items responded to correctly upon last encounter.

166

Chapter 6: Testing and drilling
NameRandom glosses
Visual similarity
Reading alternations
Random surfaces
Random readings
Total

Type
# questions
Simple11580
57.1%
Adaptive 2567
12.7%
Adaptive 2346
11.6%
Simple1948
9.6%
Simple1831
9.0%20272 100.0%

Table 6.5: Plugin name, type and number of questions generated.
users already had good knowledge of their syllabus, perhaps oﬀering an explanation why
they too did not show the same level of improvement, despite heavy use of the system.

Measuring adaptiveness
Kanji Tester is designed to generate authentic tests in two main ways: ﬁrstly, we makeuse of error models borrowed from the new FOKS system (Chapter 5). Secondly, we update these error models with user responses, allowing them to adapt the test to each user
individually. As discussed in Section 6.3, Kanji Tester maintains two error models for eachindividual, namely a kanji reading model and a kanji misrecognition model, both of which
are updated continuously with the user’s responses. Recall that each user’s tests alternate
between control and adaptive versions, both of which consist of randomly generated questions. Control and adaptive tests consist of multiple choice questions of the same type, eachwith distractors selected randomly from the same pool. However, for control questions the
distractors are sampled from the pool with uniform probability; for adaptive questions the
distractors are sampled according to the likelihood of eliciting a user error predicted by theuser’s error model.
e control/adaptive distinction is designed to allow us to better evaluate the utilityof having adaptive user models, and by having both forms of testing draw on the same
distractor pools we set a strong baseline from which to improve. In this section, we examine
the types of questions used and use the control/adaptive question distinction to measure theextent and utility of our adaptive user models.
Recall that each question in a test is generated by a plugin, with each plugin responsible

Chapter 6: Testing and drilling

167

Item type
Mean # exposures
Words
12.0
Kanji
12.4
Kanji combined
45.7
Table 6.6: Mean number of exposures across all users by item type. e Kanji Combined
type counts words containing kanji as exposures for those kanji.
for a limited range of question types. ere are two types of plugins, namely simple plugins
which do not adapt to the user, and adaptive plugins which maintain an error model which
adjusts to user responses. Our plugins use the terms gloss to refer to a word’s translation in
English, surface to refer to a word’s visual form, and reading to refer to its pronunciation.
Table 6.5 shows the distribution of questions generated by each question plugin. It
is quickly apparent that the majority of questions are generated by the “Random glosses”
plugin, covering some 57.1% of questions. is occurs because many words, especially atearlier levels of proﬁciency, are not yet represented by their full kanji form. is prevents
us from using our more sophisticated adaptive question types, which centre around kanji
misreading or misrecognition. In these cases, only the reading-meaning relationship could
be tested, hence the large coverage of this plugin in particular. is is an immediate blow toadaptivity, since the adaptive plugins need user responses in order to tune the user models.
Now, our two adaptive user models are (reading|kanji) and (kanji′ |kanji). In order forthese models to adapt to an individual user, they must face questions generated using these
distributions, i.e. generated by their corresponding adaptive plugins. However, Table 6.6
shows that across all users, each word has only 12 exposures on average, kanji not muchhigher at 12.4. Since each syllabus has many words and kanji, this means that on averageeach user won’t encounter each word or kanji more than once. is is similarly a big problem
for adaptivity at the per-user per-kanji level, although the situation is slightly improved
because kanji may occur in many words, as the Kanji combined ﬁgure shows.
Given these ﬁgures, we should expect very little adaptivity from Kanji Tester in its generation of questions. However, we can nonetheless measure eﬀectiveness of the adaptive
plugins in comparison to their simple counterparts by comparing adaptive plugins to theirsimple counterparts. Recall that for each user, every second test completed was a control test
which used simple plugins only. Simple plugins used the same error candidates, but ignored

168

Chapter 6: Testing and drilling

their probability distribution and simply used a uniform distribution across candidates.
Figure 6.22 provides error rates for each plugin, and shows that questions based on
surface form are the easiest, whereas those based on reading are the hardest. Comparing
the simple and adaptive plugins, the gap between the two reading question plugins is substantial. Using a two-tailed Student’s t-test between error vectors shows the diﬀerence issigniﬁcant to the 99% level. is suggests that learners make many more reading errors
when the distractors are plausible misreadings. On the other hand, simple and adaptive
surface questions are roughly comparable in error rate, with their diﬀerence signiﬁcant only
to a 20% level. is could indicate either that learners make few errors of this type, or that
Adaptive
Pluginour priors are
too type
weak forSimple
these models.
Note that questions based on gloss have no adapReading

0.14

0.19

tive counterpart,error model
could be easily constructed which could
Surface since no appropriate
0.09
0.09
Gloss

0.12

guarantee that distractor glosses were actually incorrect.
Error rate by plugin type
0.20

0.15

0.10

0.05

0

Reading

SimpleSurface

Gloss

Adaptive

Figure 6.22: Error rates compared for simple and adaptive plugins.
In general it appears that adaptive questions are able to generate more user errors compared to their simple counterparts, but how was this possible if they adapted little to eachuser? e more diﬃcult adaptive questions are generated because the user models use rebuilt
FOKS error models for priors. Since per-user error models adapt little, they are in generalChapter 6: Testing and drilling169

similar across all users, but still eﬀective in generating distractors due to these priors.

Error models
We used two main error models: our reading alternation model (reading|kanji), and ourgrapheme alternation model (kanji′ |kanji). A signiﬁcant motivation for constructing Kanji
Tester was the opportunity to test these models proactively rather than reactively. FOKS
attempts to recover from user queries, but if users are conservative in their querying, we
may encounter only limited forms of errors. Since Kanji Tester is a more constrained environment, we can attempt to provoke errors by introducing distractors to users, and thus
obtain greater coverage over error forms. is greater coverage comes at a cost: users ofdictionaries have at least partial knowledge of the word, whereas the worst-case scenario fora test question is where the user has no applicable knowledge, and guesses near-randomly.
Nonetheless, the diﬀerence in error patterns between the two applications is worth examining.
We ﬁrst consider the reading alternation model, which provoked signiﬁcantly more errors than average. Using the same query explanation feature of FOKS, we can recover thetypes of errors made by users. Table 6.7 provides the distribution of error types users made,and is an interesting comparison with our earlier analysis of FOKS queries (see Table 5.7).
Whilst choice of reading remains the most common error type, non-compositional reading errors are signiﬁcantly reduced. is is most likely because the syllabi available fortesting contain few proper nouns, which are often non-compositional. e other main difference is the increase in voicing errors compared to vowel length errors. Although somevoicing errors were cases of sequential voicing, such as choosing tekami as the reading for
手紙 tegami “letter”, many also mistakenly voiced initial constants, for example choosing
dzuki as the reading for 月 tsuki “month”.
ere are two plausible reasons why these additional voicing errors are being made bylearners. Firstly, they may have encountered kanji or words in larger compounds ﬁrst, where
sequential voicing was correctly applied. From these encounters, they may assume mistakenly that the reading of the item outside the compound is also voiced this way. For example,
if the learner knows the place name 新橋 shiNbashi, they may mistakenly read 橋 “bridge” as170

Chapter 6: Testing and drilling
Error type
Choice of reading
Voicing error
Vowel length
Non-compositional reading
Sound euphony
Palatalisation
Total

Frequency
174 76.0%
23
10.0%
14
6.1%
9
3.9%
8
3.5%
1
0.4%
229 100.0%

Table 6.7: Error types for adaptive reading questions.
Word/kanji
急度
ki o
出る
deru
意
i
長い
nagai
題
dai
代
dai
仕方
shikata
小父さん ojisaN“without fail”“to appear”“feelings”“long”“subject”“price”“way”“uncle”# Errors
7
6
5
4
4
4
4
4

Users chose
急言/多計/意文/急来/思度/悪言
千る/高る/南る/者る/計る思/春
大い/英い/足い
員/貸/買
貸/会/仕
生方/仕走/工去/行方
小入さん/少父さん/小大さん

Table 6.8: Top 8 words or kanji by number of grapheme substitution errors.

bashi instead of its correct reading hashi. Secondly, they may be confused by the graphemic
similarity between pronunciation strings, which are provided in kana. In Japanese, voicing
is marked in hiragana and katakana by a diacritic. For example, ha, ba and pa (unvoiced,
voiced, semi-voiced) are written as は , ば and ぱ . Since these diacritics are small, learners
could easily visually mistake voiced and unvoiced variants of these kana. Such errors are notexplicitly modelled as reading alternations; incorporating knowledge of them into futuresystems would prove interesting.
We now examine grapheme substitution errors. Table 6.8 shows the most common
errors made by users of Kanji Tester. On the right of each word, we see distractors which
diﬀerent users mistook for the correct answer. From these results, we can see mixed levelsof apparent similarity between the user errors and the correct answer. For example, pairs
such as 小父さん and 小大さん seem visually similar, yet others such as 出る and 者る
do not. e presence of the latter type of pair suggests that in some cases, the user simplyChapter 6: Testing and drilling171

guessed randomly amongst alternatives.
is may have happened in our reading results as well, but each reading result is alsoa plausible misreading, whereas noise in our graphemic distance models means that someneighbours generated by the model will share little visual similarity. An alternative explanation of the same problem is that some regions in grapheme space are visually dense, andothers visually sparse. Sparse items have few plausible neighbours, yet we must still generate a suﬃciently large distractor pool for such items for use in testing. us, even a very
accurate graphemic distance model is forced to occasionally include neighbours beyond aplausible misrecognition threshold.

6.5

Discussion

In this chapter we have discussed a system allowing users to test their knowledge ofJapanese vocabulary, within the scope of two supported syllabi. e goal of the system wasto allow repeatable, accurate self-evaluation by replicating limited aspects of human tests.
Have we succeeded?
We can measure success in several ways. We have clearly built a system that allows
rapid self-evaluation, assured by our random sampling over the user’s chosen syllabus andquestion generation which is at least minimally resistant to random guessing. However,any system with these features would achieve this aim without requiring signiﬁcant error
modelling. On the other hand, have we replicated human tests?
Our reading alternation questions are certainly successful, eliciting many more errors
than their control counterparts. Our grapheme alternation questions however were comparable to their control counterparts, and thus do not match a human generated standard.
ere are several problems we face generating such questions. Firstly, our best graphemic
distance model – stroke edit distance – still has signiﬁcant noise when compared with human judgements. is noise must be reduced if we are to generate questions with morediﬃcult distractors. Secondly, the natural sparsity of some areas in the visual similarity
space means that there are simply not enough motivated distractors to generate diﬃcult
questions for all kanji or all words. is is solved in human-generated tests by the use ofplausible non-kanji, made up of real components in combinations which do not occur in

172

Chapter 6: Testing and drilling

real-life. Ultimately, the only way to solve the data sparsity problem is to ﬁnd a way toautomatically generate distractors using such non-kanji; as yet no such automated system
exists, though use of Chinese, Korean or archaic variants could alleviate this concern.
We drew a distinction between diﬀerent forms of testing in Section 6.1, from ﬂashcards
to human-generated tests, and claimed that Kanji Tester would provide the availability ofﬂashcards, yet with validity and scope closer to human tests. In its current form, Kanji
Tester serves as a useful intermediary between these two forms of testing. Since it is not
yet as powerful as human-generated tests, it cannot replace them. Yet neither can it replace
ﬂashcards, for the simple reason that ﬂashcards provide a progression through the syllabus
which facilitates learning. In comparison to ﬂashcards, Kanji Tester is superior in the levelof self-assessment, yet inferior as a drilling tool. Could this be improved?
Whilst ﬂashcards will continue to be useful, a future system could certainly beneﬁt bysimply providing a linear progression through the syllabus, returning repeatedly to items responded to incorrectly. For example, it could use Leitner’s (1972) spaced repetition method,
as discussed in Section 3.1. is linear progression would ensure full eventual coverage ofthe user’s syllabus. However it would not be useful for testing the user’s vocabulary knowledge on the whole syllabus. For that, the current random sampling of the syllabus gives
better validity. Although the current method of testing ability has not been explicitly evaluated against current forms of paper testing. It cannot either be used by teachers to makedecisions about their students without such formal comparisons.
For these reasons, the suggested future system would thus provide two modes of testing.
One which provided an alternative to ﬂashcards, as a means of vocabulary study. e other,purely as a means of vocabulary self-assessment, yet evaluated formally against other tests
to prove its validity. is formal assessment would allow it to bridge the divide between
learner tools for self-use, and tools to aid language educators in rapid assessment of theirstudents.
Overall, the current Kanji Tester system provides a compelling example of increasing
possibility of generating intelligent learning tools for learners, based on strongly motivatederror modelling. e better we can understand learners, the better we can adapt to theirneeds. Applied linguists have long looked at the minutiae of learner errors to better understand how they acquire language; our work does the same, but uses those errors to support

Chapter 6: Testing and drilling173

learners in their self-study and self-discovery of language.

6.6

Conclusion

Nearly all forms of language learning can be construed as testing. In this chapter we havebound together all aspects of our work and embedded them in a vocabulary testing application, Kanji Tester, which approximates the well known JLPT test family in its coverage andtesting of vocabulary. Kanji Tester makes use of both intelligent kanji (mis)reading models
from the FOKS dictionary (Chapter 5), and a new kanji (mis)recognition model generated
from the novel stroke edit distance metric (Chapter 4). Furthermore, we explored how
these error models might change over time in response to user input. In particular, theadaptive reading model achieved a statistically signiﬁcant increase in the number of user
errors elicited.
e data sparsity issues we encountered suggest two possible directions the existing system could take in the future. Firstly, Kanji Tester could attempt to move to a user modelof coarser granularity, for example by pooling user models for learners from the same ﬁrst
language background. Secondly, Kanji Tester could divide itself into two applications: adrilling application, which would revisit stimulus according to a spaced repetition schedule instead of randomly, and a testing application, meant for occasional authoritative testing. is would better match the patterns of current use, and by revisiting incorrect items
more frequently in drill mode data sparsity issues would be alleviated. Fortunately, thecurrent body of user responses serves as a useful data set on which to evaluate new and as
yet untested user models, without having to deploy them and collect new user data overtime. Over longer time periods, it may also serve as a useful data set for measuring learner
improvement.
More broadly, Kanji Tester has limited itself to vocabulary testing and focused on kanjibased errors. ere are many additional aspects to proﬁciency which could be tested, anda template-based systems such as Zock and Afantenos’s (2007) pattern drills seem a good
partner for our approach. A current shortcoming of Kanji Tester is its lack of intelligent
gloss-based questions. A large barrier to such questions currently lies in the need to avoid
using too-near-synonyms as distractors, since their use poses the risk of unintentionally174

Chapter 6: Testing and drilling

generating questions with more than one valid answer. If this risk could be avoided throughuse of semantic similarity measures, stronger testing would be possible.

Chapter 7
Conclusions
7.1

Conclusions

is thesis has investigated the use of linguistic error modelling to support language
learners in their autonomous self-study of Japanese vocabulary. Acquiring suﬃcient vocabulary is perhaps the most diﬃcult task in language learning because the breadth of knowledge to be acquired is enormous, too much to be taught explicitly. e most commonly
advocated solution to acquiring large amounts of vocabulary is to read widely, yet Nuttall(1996) found a system of positive feedback not only in successful reading, but in unsuccessful reading too. is indicates some sort of tipping or boundary point beyond which
successful readers propel themselves towards ﬂuency, and before which reading is of limited use and can even reduce motivation. Laufer (1997) suggests this tipping point lies atthe 3000 word family mark for vocabulary knowledge. e central problem in vocabulary
learning is thus to allow learners to reach this mark, and to support their early attempts at
reading.
is thesis has made three main contributions towards solving this problem. Firstly,
noting the potential for lexical relationships to support vocabulary growth, it examined in
depth graphemic relationships between words and characters based on similarity of visual
form, and compared several novel similarity metrics for Japanese kanji. In order to do this,it has also generated three useful data sets for the evaluation of such models, not including
the additional data which will continue to be generated over time by both the FOKS and
175

176

Chapter 7: Conclusions

Kanji Tester systems as they are used. In its evaluation, it has found the stroke edit distanceand tree edit distance metrics to most eﬀectively match human judgements, and based on
the available data has identiﬁed stroke edit distance as the preferred distance metric due to
its relative eﬃciency in comparison to its tree-based cousin.
e second main contribution involved the transformation of stroke edit distance into a
confusability model for Japanese kanji, and the incorporation of this model into an existing
dictionary to support search-by-similarity. Several other minor enhancements were madeto the FOKS dictionary as part of this rebuild, including the addition of a more scalable
grapheme-phoneme alignment algorithm. Upon post-hoc analysis of log data, our new
search method was found to have been quickly adopted by users, to the extent that searches
by similarity outnumbered the previous form of intelligent reading search during the samelog period, indicating their utility.
Finally, we combined existing phonemic and graphemic error models applied these to
the new area of adaptive testing in the form of the Kanji Tester site. By emulating theauthoritative JLPT test, Kanji Tester aimed to allow users better self-evaluation throughrapid and repeatable proﬁciency testing. rough analysis of user responses over a several
month log period, we established that use of adaptive testing increased the diﬃculty of test
questions signiﬁcantly over a control baseline.
ese theoretical and practical contributions combined to form a platform for secondlanguage vocabulary growth in Japanese.

Graphemic similarity
Four new graphemic distance models were proposed for Japanese kanji, based on radicals, pixels, tree-based structure and strokes. ese distance models were evaluated on threemain data sets: explicit human similarity judgements, expert judgements from a commercial
ﬂashcard set, and native speaker judgements from a candidate pool experiment.
e ﬁrst experiment showed that for low-to-medium similarity pairs, shared radicals
and similar layout were most important for measuring similarity. Furthermore, the results
suggested that non-speakers of Japanese do not diﬀer fundamentally in their judgements
to native speakers, but rather their judgements gain in consistency as their experience with

Chapter 7: Conclusions

177

such characters increases. e ﬂashcard data set consisted of expert-selected high-similarity
pairs, and here radical methods performed the most poorly across all methods of measurement. Instead, stroke and tree-based methods modelled most closely the expert judgements,
followed by pixel comparison of kanji images. A distractor pool experiment eliciting native
speaker judgements conﬁrmed this ordering, suggesting that for high-similarity pairs, bothform and layout of components were important. e success of the stroke-based metric was
explained in terms of its success in fuzzy matching of structural and form-based features,and due to its eﬃciency in comparison to other metrics, it was established as the preferred
metric given our data.

Dictionary search
We examined in detail how an error-correcting dictionary, FOKS, could be extended
and augmented to increase its accessibility and thus its support for early reading. As part ofa complete rebuild, we modiﬁed the dictionary’s grapheme-phoneme alignment algorithm,
showing that use of a kanji reading resource allowed far faster alignment times throughstronger alignment constraints. Furthermore, its TF-IDF scoring function was shown toperform better when reduced to the IDF component alone, due to the importance of segmenting maximally in the GP alignment task.
We extended the dictionary’s coverage of place names by making use of a simple gazetteer
mined from Japan Post, and improved the display of translations to accommodate richer
word information and yet remain manageable for users. We then made FOKS’s internal
error models transparent to users by providing a query explanation tool, which also serves
to explain the derivation of correct kanji compound readings.
Finally, and most signiﬁcantly, we incorporated our stroke-based grapheme distancemetric into the dictionary to form a new error-correcting search by visual form. is innovative form of search allows users to search for visually similar words using their known
neighbours. For example, a user could use the query 補左 to ﬁnd the word 補佐 hosa “help”,
based on the similarity between 左 and 佐. Query logs showed that this new form of searchis actively used, and is thus providing beneﬁt for learners.

178

Chapter 7: Conclusions

Vocabulary testing
We demonstrated how linguistic error models can be transferred from the dictionary
search domain to the language testing domain, where they can actively provoke user errors
rather than passively correcting for them. e reading and graphemic error models from therebuilt FOKS system were used to develop Kanji Tester, an adaptive testing system which
generates multiple choice questions which are potentially unique for every user and every
test.
We discussed various methods for grouping users and errors at diﬀering levels of granularity, ultimately choosing to model users as individuals and errors at the kanji level. We
then proposed an update rule for this form of error modelling, and applied this rule to
make tests adapt to user responses, increasing the likelihood of distractors which previously
provoked an incorrect response.
Finally, we performed extensive log analysis to determine the relationship between useof the system and user scores on the system, which serve as estimates of receptive vocabularyknowledge. On the whole we found a weak positive correlation between time and test scores,suggesting that users of the system improved in ability the longer they used the system. We
also compared adaptive and control question types, and found that adaptive questions caused
more errors than control questions, especially for the reading error model. Data sparsenesslimited the adaptivity of error models to users, however strong prior distributions meantthat tests were suﬃciently diﬃcult nonetheless.

7.2

Future work

Many open issues remain for future investigation. e following sections describe theseissues, aligning them roughly to our main areas of contribution.

Lexical relationships
In this thesis we focused on graphemic similarity as a novel and under-utilised relationship between words, and measured this similarity through a range of distance metrics
which we evaluated. Our best performing metric, stroke edit distance, still contains sig-

Chapter 7: Conclusions

179

niﬁcant noise which impedes better lookup and testing. However, this area of study is nolonger data poor. We have contributed ﬁve data sets for evaluating graphemic similarity:
two experiments, ﬂashcard distractors, and two sets of log data. ese provide a strong
foundation for the development of better performing graphemic distance metrics.
A simple direction to consider is the appropriate costs to use for tree and stroke edit
distance calculations; we used unit costs, but if some features prove to be far more important
than others, variable costs might provide a better result. More broadly, there is also scope
to examine a broader range of OCR techniques for Japanese kanji, and to develop distancemetrics from these techniques.
Any such metric used generates a topology on the space of kanji. Interesting aspectsof these topologies should be explored, and explained with reference to models of kanji
perception. For example, orthographic density about a word has already been explored in
psycholinguistics; this could be extended to examine and explain diﬀerences between low
and high-density regions of orthographic space.
e limits of visual neighbour accessibility should be explored to determine the drop-oﬀ
in neighbour availability as kanji grow more visually distant from one another. Visual neighbours that are close enough can be considered linked; the resulting network may have interesting properties worth exploring, and comparing with Yamamoto and Yamazaki’s (2009)
recent work on network properties of kanji compounds. ere are also other graphemic relationships other than similarity, for example containment. ese alternative relationships
may prove more accessible to learners than similarity, or at the very least complementary,and should thus be examined.
Beyond graphemic relationships, there is much work to be done in mapping out, making
sense of and unifying the various forms of semantic relationships, since doing so will greatly
aid lexicography, and will provide meaningful new methods of understanding words andtheir relationships to each other.

Dictionary search
In this thesis we took an existing dictionary, FOKS, and rebuilt it to increase its accessibility, however we believe there remain improvements worth investing in. Some of these

180

Chapter 7: Conclusions

opportunities lie in correcting known error types, such as errors due to mistaken voicing ofkana; others are in helping learners understand the mistakes they have made, for example
by providing a spatial explanation of how two similar kanji diﬀer.
We have already mentioned the need for improved graphemic distance models, but anyother form of lexical relationship can be used to augment a dictionary in useful ways. For
example, semantic relatedness could be used to provide nearby words to users for comparison.
e most interesting area to pursue is the ability to provide users with the means toextend the dictionary in useful ways, thus “crowdsourcing” dictionary enhancements. eTatoeba project1 already does this for example sentences, so that representative examples
of word usage can be provided by users themselves. is could be combined with semantic
similarity measures between words, so that very similar words could have example sentences provided which demonstrate the diﬀerence between them. Choosing between nearsynonyms is extremely diﬃcult for learners, since they lack contextual knowledge about
when each word is appropriate, and what additional meaning a word may connote. An
appropriate dictionary extension would help solve this diﬃcult problem, and would thus
alleviate the diﬃculties associated with lexical gridding issues between a learner’s ﬁrst andsecond languages. ere have been several other recent attempts to render dictionaries far
more useful and useable, for example, use of semagrams as a rich semantic encoding ofwords in dictionaries (Moerdijk et al. 2008). Tapping into user populations to crowdsource
this additional richness would also be an approach worth exploring.
An alternative approach for improving dictionary accessibility would be to conduct formal user studies of dictionary search, for example using task timing and verbal protocols(Ericsson and Simon 1985) to elicit hard data on dictionary lookup performance. Suchdata could provide a principled means of discriminating between existing dictionary oﬀerings based on accessibility, or could be used to look for particular points of diﬃcult which
remain for learners.

1

http://tatoeba.org/

Chapter 7: Conclusions

181

Proﬁciency testing
We have shown that adaptive vocabulary testing is possible, and argued for its futurerole in replacing human-constructed tests. Kanji Tester is an important ﬁrst step, but there
is still a signiﬁcant gap between human tests and automated tests. Several open researchareas could help close this gap.
Firstly, we know that vocabulary knowledge is graded, since there are many aspectsof word knowledge, and that productive tests are more diﬃcult than receptive tests, suchas Kanji Tester. If Kanji Tester incorporated both forms of test question, the distinction
between these two types or levels of knowledge could be analysed and solidiﬁed throughextensive log analysis.
Secondly, having argued for the importance of vocabulary knowledge, we could attempt
to estimate the size of a user’s vocabulary knowledge directly, for example using a wordfamiliarity database based on the method of Amano and Kondo (1998). Providing thisestimate to learners would allow them to set vocabulary size goals and measure progress
towards achieving such goals.
We also noted the various levels and granularities of user and error models available foruse, but only experimented with a single of these combinations. By using the existing user
response data, we could evaluate diﬀerent user model combinations, and determine whatmodel best predicts actual user behaviour. is in turn would provide useful insights as tothe extent to which user errors and misconceptions are shared across diﬀerent cross-sectionsof the learner community.
ere is signiﬁcant scope for better receptive question generation, through new anddiﬀerent error models. Existing work on transliteration, such as that by Pervouchine et al.
(2009), could be leveraged to generate plausible distractors for katakana loan-words. Learners are also known to confuse words which are semantically similar (Mondria 2007). Semantic similarity measures, as discussed in the previous section, could thus be used to generate
error-provoking gloss distractors based on semantic similarity. For this purpose, Japanese
WordNet (Isahara et al. 2008) could be used, choosing siblings or cousins as semanticsimilarity-based distractors.
e ability to hand-write Japanese kanji is a productive skill requiring recall of form

182

Chapter 7: Conclusions

rather than pronunciation, one which we have not discussed at length in this thesis. Modern
hand-writing dictionary interfaces such as those discussed in Section 2.2 suggest a way that
this skill could be incorporated automatically into tests. Tests requiring both input of kanji
by form and input of typed kanji by pronunciation could provide interesting data on therelative productive abilities of learners with respect to form vs sound recall.
Nearly all our work on Japanese can be easily extended to Chinese, especially workon graphemic error modelling, since the Chinese writing system uses similar characters toJapanese kanji. e Pinyomi dictionary (Yencken et al. 2007) is a straightforward candidate for such improvement. Its Japanese-Chinese dictionary interface based on lookup-bytransliteration could easily support search by similar kanji or hanzi. Doing so would involve
rebuilding one of the similarity models to encompass both hanzi and kanji, but the improvement in accessibility could be signiﬁcant. Such multilingual models could also be useful inmonolingual testing, since they could alleviate the sparse visual neighbour problem.
Finally, recall that in Chapter 3 we noted our limited coverage of the Chinese- andJapanese-language literature, especially as relates to native speaker development. We encourage future multi-lingual surveys into these areas, as techniques designed to aid second
language learners may equally prove useful for native speakers, especially child learners during their development or adults facing rare words.

7.3

Summary

In summary, this thesis argued for better support for second language learners in their
vocabulary study and early reading experiences, and indeed provides support in both areas.
Firstly, it developed new linguistic error models based on graphemic proximity of words andkanji. Secondly, it extended an existing dictionary, improving its usability and accessibility through graphemic-error correcting search. Finally, it used error models to provide anadaptive testing interface for learners to self-evaluate their study progress. Together, theseapplications form a platform to assist learners in their autonomous self-study of vocabulary,
and thus in their ultimate progression towards ﬂuency.

Bibliography
Allen, Jonathan, Sheri Hunnicut, and Dennis Klatt. 1987. From Text To
Speech, e MITALK System. Cambridge, UK: Cambridge University Press.
Amano, Shigeaki, and Tadahisa Kondo. 1998. Estimation of mental lexicon sizewith word familiarity database. In Proceedings of the 5th International Conference on
Spoken Language Processing, volume 5, 2119–2123, Sydney, Australia.
Apel, Ulrich, and Julien Quint. 2004. Building a graphetic dictionary for Japanese
kanji – character look up based on brush strokes or stroke groups, and the display ofkanji as path data. In Proceedings of the COLING 2004 Workshop on Enhancing and Using
Electronic Dictionaries, 36–39, Geneva, Switzerland.
Baayen, Harald. 1993. Statistical models for word frequency distributions: A linguistic
evaluation. Computers and the Humanities 26.347–363.
Backhouse, A. E. 1996. e Japanese Language: An Introduction. Melbourne, Australia:
Oxford University Press.
Baddeley, A. D. 1997. Human memory: eory and Practice. Boston: Allyn and Bacon,
revised ed edition.
Baddeley, Alan. 2003. Working memory and language: an overview. Journal ofCommunication Disorders 189–208.
Baldwin, Timothy, and Hozumi Tanaka. 1999a. e applications of unsupervised
learning to Japanese grapheme-phoneme alignment. In Proceedings of the 1999 ACL
Workshop on Unsupervised Learning in Natural Language, 9–16, College Park, USA.
——, and ——. 1999b. Automated Japanese grapheme-phoneme alignment. In Proceedingsof the International Conference on Cognitive Science, 349–354, Tokyo, Japan.
——, and ——. 2000. A comparative study of unsupervised grapheme-phoneme alignment
methods. In Proceedings of the 22nd Annual Meeting of the Cognitive Science Society,597–602, Philadelphia, USA.
183

184

BIBLIOGRAPHY

Barker, David. 2007. A personalized approach to analyzing ‘cost’ and ‘beneﬁt’ invocabulary selection. System 4.523–533.
Begg, Andy. 1997. Some emerging inﬂuences underpinning assessment in statistics. In
e Assessment Challenge in Statistics Education, ed. by I. Gal and J. B. Garﬁeld, chapter 2,
17–25. Amsterdam, Netherlands: IOS Press.
Bell, Timothy. 1998. Extensive reading: Why? and how? e Internet TESL Journal
4.
Bilac, Slaven, 2005. Automatically extending the dictionary to increase its coverage andaccessibility. Tokyo Institute of Technology dissertation.
——, Timothy Baldwin, and Hozumi Tanaka. 1999. Incremental Japanesegrapheme-phoneme alignment. In Information Processing Society of Japan SIG Notes,
volume 99-NL-209, 47–54.
——, ——, and ——, 2003. Modeling learners’ cognitive processes for improved dictionary
accessibility. Handout at 10th International Conference of the European Association forJapanese Studies, Warsaw, Poland.
——, ——, and ——. 2004. Evaluating the FOKS error model. In Proceedings of the
4th International Conference on Language Resources and Evaluation, 2119–2122, Lisbon,
Portugal.
——, and Hozumi Tanaka. 2005. Direct combination of spelling and pronunciation
information for robust back-transliteration. In Proceedings of the 6th International Conference on Computational Linguistics and Intelligent Text Processing, 413–424, Mexico
City, Mexico.
Bille, Philip. 2005. A survey on tree edit distance and related problems. eoreticalComputer Science 337.217–239.
Black, Alan W., Paul Taylor, and Richard Caley, 1999. e Festival speechsynthesis system. System documentation.
Breen, Jim. 1995. Building an electronic Japanese-English dictionary. In Proceedings ofthe 1995 Japanese Studies Association of Australia Conference, Brisbane, Queensland.
Brown, Peter F., Stephen A. Della Pietra, Vincent J. Della Pietra,
and Robert L. Mercer. 1993. e mathematics of statistical machine translation:
Parameter estimation. Computational Linguistics 19.263–311.
Brown, Roger, and David McNeill. 1966. e “tip of the tongue” phenomenon.
Journal of Verbal Learning and Verbal Behavior 5.325–337.

BIBLIOGRAPHY

185

Brusilovsky, Peter. 2001. Adaptive hypermedia. User Modeling and User-AdaptedInteraction 11.87–110.
Bull, Joanna, and Colleen McKenna. 2004. Blueprint for Computer-Assisted Assessment. London, UK: Routledge Falmer.
Carreiras, Manuel, Manuel Perea, and Jonathan Grainger. 1997. Eﬀects oforthographic neighborhood in visual word recognition: Cross-task comparisons. Journal of Experimental Psychology: Learning, Memory and Cognition 23.857–871.
Chalhoub–Deville, Micheline, and Craig Deville. 1999. Computer adaptive
testing in second language contexts. Annual Review of Applied Linguistics 19.273–299.
Clark, Stephen, and James R. Curran. 2004. e importance of supertagging for
wide-coverage CCG parsing. In Proceedings of 20th International Conference on Computational Linguistics, 282–288, Geneva, Switzerland.
Coady, James. 1997. L2 vocabulary acquisition through extensive reading. In Second
Language Vocabulary Acquisition, ed. by James Coady and omas Huckin, 225–237.
Cambridge, UK: Cambridge University Press.
Cohen, William W., Pradeep Ravikumar, and Stephen E. Feinberg. 2003.
A comparison of string distance metrics for name-matching tasks. In Proceedings of the
IJCAI-2003 Workshop on Information Integration on the Web, 73–78, Acapulco, Mexico.
Craik, Fergus I. M., and Endel Tulving. 1975. Depth of processing and theretention of words in episodic memory. Journal of Experimental Psychology: General
104.268–294.
Cruse, D. A. 1986. Lexical semantics. New York: Cambridge University Press.
Demaine, Erik D., Shay Mozes, Benjamin Rossman, and Oren Weimann.
2007. An optimal decomposition algorithm for tree edit distance. In Proceedings ofthe 34th International Colloquium on Automata, Languages and Programming, 146–157,
Wrocław, Poland. Springer.
Dycus, David. 1997. Guessing word meaning from context: Should we encourage it?
Literacy Across Cultures 1.1–6.
Ericsson, K. Anders, and Herbert A. Simon. 1985. Protocol Analysis: Verbal
Reports as Data. Cambridge, Massachusetts: MIT Press.
Espinosa, María Paz, and Javier Gardeazabal. 2007. Optimal Correction for
Guessing in Multiple-Choice Tests. Technical report, Department of Foundations of
Economic Analysis II, University of the Basque Country.

186

BIBLIOGRAPHY

Eugenio, Barbara Di, and Michael Glass. 2004. e kappa statistic: A second
look. Computational Linguistics 30.95–101.
Fellbaum, Christiane. 1998. Wordnet: An Electronic Lexical Database. Cambridge,
MA, USA: MIT Press, 1st edition.
Ferret, Olivier, and Michael Zock. 2006. Enhancing electronic dictionaries withan index based on associations. In Proceedings of the 21st International Conference on
Computational Linguistics and the 44th annual meeting of the ACL, 281–288, Sydney,
Australia.
Flores d’Arcais, Giovanni B., Hirofumi Saito, and Masahiro Kawakami.
1995. Phonological and semantic activation in reading kanji characters. Journal of
Experimental Psychology: Learning, Memory and Cognition 21.34–42.
Fraser, Carol A. 1999. Lexical processing strategy use and vocabulary learning
through reading. Studies in Second Language Acquisition 21.225–241.
Gaskell, M. Gareth, and Nicolas Dumay. 2003. Lexical competition and theacquisition of novel words. Cognition 89.105–132.
Gaume, Bruno, Karine Duvignau, Laurent Prévot, and Yann Desalle.
2008. Toward a cognitive organization for electronic dictionaries, the case for semantic proxemy. In Proceedings of the workshop on Cognitive Aspects of the Lexicon, 86–93,
Manchester, UK.
Goh, Chooi-Ling, Masayuki Asahara, and Yuji Matsumoto. 2005. Building a Japanese-Chinese dictionary using kanji/hanzi conversion. In Proceedings of the
2nd International Joint Conference on Natural Language Processing, 670–681, Jeju Island,
Korea.
Halpern, Jack (ed.) 1999. e Kodansha Kanji Learner’s Dictionary. Tokyo, Japan:
Kodansha International.
Hambleton, Ronald K., and Russell W. Jones. 1993. Comparison of Classical
Test eory and Item Response eory and their applications to test development.
Educational Measurement: Issues and Practice 12.38–47.
Handke, Jürgen. 1995. e Structure of the Lexicon: Human vs Machine. Berlin: Mouton
de Gruyter.
Harris, Zellig S. 1954. Distributional structure. Word 10.146–162.
Heift, Trude, and Devlan Nicholson. 2001. Web delivery of adaptive and interactive language tutoring. International Journal of Artiﬁcial Intelligence in Education
12.310–324.

BIBLIOGRAPHY

187

Heisig, James W. 1985. Remembering the Kanji I: A Complete Course on how Not toForget the Meaning and Writing of Japanese Characters. Tokyo, Japan: Japan Publication
Trading Co.
Hsueh-Chao, Marcella Hu, and Paul Nation. 2000. Unknown vocabulary density
and reading comprehension. Reading in a Foreign Language 13.403–430.
Huckin, Thomas, and James Coady. 1999. Incidental vocabulary acquisition in asecond language. Studies in Second Language Acquisition 21.181–193.
Ikehara, Satoru, Masahiro Miyazaki, Satoshi Shirai, Akio Yokoo, Hiromi
Nakaiwa, Kentaro Ogura, Yoshifumi Ooyama, and Yoshihiko Hayashi.
1997. Goi-Taikei – A Japanese Lexicon. Tokyo, Japan: Iwanami Shoten.
Isahara, Hitoshi, Francis Bond, Kiyotaka Uchimoto, Masao Utiyama, andKyoko Kanzaki. 2008. Development of the Japanese wordnet. In Proceedings of the
6th International Language Resources and Evaluation, Marrakech, Morocco.
Japan Foundation, 2009. What is the JLPT: Contents of the Test. Available at http://www.jlpt.jp/e/about/content.html. Accessed 7th June 2009.
Jin, Zhihui. 2008. Pinyomi-lite - improved Japanese-Chinese dictionary lookup using
logograph transliteration. In Proceedings of the 14th Annual Meeting of the Japan Society
for Natural Language Processing, Tokyo, Japan.
Joyce, Terry. 1999. Lexical access and the mental lexicon for two-kanji compoundwords: A priming paradigm study. In Proceedings of the 7th International Conference on
Conceptual Structures, 1–12, Blacksburg, VA, USA.
——. 2002. Constituent-morpheme priming: Implications from the morphology of twokanji compound words. Japanese Psychological Research 44.79–90.
——. 2005. Constructing a large-scale database of Japanese word associations. Glottometrics
10.82–98.
Knight, Kevin, and Jonathan Graehl. 1998. Machine transliteration. Computational Linguistics 24.599–612.
Koda, Keiko. 2007. Reading and language learning: Crosslinguistic constraints on
second language reading development. Language Learning 57.1–44.
Kondrak, Grzegorz. 2003. Identifying complex sound correspondences in bilingual
wordlists. In Proceedings of 4th International Conference on Computational Linguistics andIntelligent Text Processing, 432–443, Mexico City, Mexico.

188

BIBLIOGRAPHY

Krashen, Stephen. 1989. We acquire vocabulary and spelling by reading: Additional
evidence for the input hypothesis. Modern Language Journal 73.440–464.
Landauer, T. K., and L. A. Streeter. 1973. Structural diﬀerences between common
and rare words: Failure or equivalence assumptions for theories of word recognition.
Journal of Verbal Learning and Verbal Behavior 12.119–131.
Landauer, Thomas K., and Susan T. Dumais. 1997. A solution to Plato’s problem: e latent semantic analysis theory of acquisition, induction and representation ofknowledge. Psychological Review 104.211–240.
Laufer, Batia. 1997. e lexical plight in second language reading; words you don’tknow, words you think you know, and words you can’t guess. In Second Language Vocabulary Acquisition, ed. by James Coady and omas Huckin, Cambridge Applied Linguistics, 20–34. Cambridge, UK: Cambridge University Press.
——, and Zahava Goldstein. 2004. Testing vocabulary knowledge: Size, strength andcomputer adaptiveness. Language Learning 54.399–436.
——, and Tamar Levitzky-Aviad. 2006. Examining the eﬀectiveness of ‘bilingual
dictionary plus’ – a dictionary for production in a foreign language. International Journalof Lexicography 19.135–155.
——. 1991. Similar Lexical Forms. Tübingen, Germany: Gunter Narr Verlag Tübingen.
Leitner, S. 1972. So lernt man lernen. Freiburg, Germany: Herder.
Levelt, Willem J. M., Ardi Roelofs, and Antje S. Meyer. 1999. A theory oflexical access in speech production. Behavioral and Brain Sciences 1–75.
Lupker, Stephen J. 2005. Visual word recognition: eories and ﬁndings. In e
Science of Reading: A Handbook, ed. by Margaret J. Snowling and Charles Hulme, chapter 3. Carlton, VIC, Australia: Blackwell Publishing.
Marso, Ronald N. 1969. e inﬂuence of test diﬃculty upon study eﬀorts and achievement. American Educational Research Journal 6.621–632.
McClelland, James L., and David E. Rumelhart. 1981. An interactive activation model of context eﬀects in letter perception, Part 1: An account of basic ﬁndings.
Psychological Review 88.375–407.
Meara, Paul. 2009. Connected Words: Word associations and second language Vocabularyacquisition. John Benjamins Publishing Company.
Miller, Georgia E. 1941. Vocabulary building through extensive reading. e English
Journal 30.664–666.

BIBLIOGRAPHY

189

Moerdijk, Fons, Carole Tiberius, and Jan Niestadt. 2008. Accessing the
ANW dictionary. In Proceedings of the 2008 Workshop on Cognitive Aspects of the Lexicon,
18–24, Manchester, UK.
Mohseni-Far,
Mohammad.
2008.
In search of the best technique for vocabulary acquisition.
Estonian Papers in Applied Linguistics(Eesti rakenduslingvistika uhingu aastaraamat) 4.121–138.
Mondria, Jan-Arjen. 2003. e eﬀects of inferring, verifying and memorizing on theretention of L2 word meanings. Studies in Second Language Acquisition 25.473–499.
——. 2007. Myths about vocabulary acquisition. Babylonia 2.63–68.
——, and Siebrich Mondria-De Vries. 1994. Eﬃciently memorizing words with thehelp of word cards and hand computer: eory and applications. System 22.47 – 57.
Murphy, Kevin R., and Charles O. Davidshofer. 1998. Psychological Testing:Principles and Applications. Upper Saddle River, NJ, USA: Prentice Hall, 4th edition
edition.
Nation, I. S. P. 2001. Learning Vocabulary in Another Language. Cambridge, UK:
Cambridge University Press.
Nation, P. 2006. Vocabulary: Second Language, 448–454. Oxford: Elsevier, 2nd ededition.
National Virtual Translation Center, 2007.
Language learning diﬃculty for English speakers. http://www.nvtc.gov/lotw/months/november/learningExpectations.html. Accessed 19th Jan 2008.
Nishiguchi, Koichi. 1994. Kanji in Context Workbook Volume 1: Book 1. Tokyo, Japan:
Japan Times.
Nuttall, Christine. 1996. Teaching Reading Skills in a Foreign Languages. London,
UK: Heinemann English Language Teaching, 2nd edition.
Pagel, Vincent, Kevin Lenzo, and Alan W. Black. 1998. Letter to sound rules
for accented lexicon compression. In Proceedings of the 5th International Conference on
Spoken Language Processsing, 252–255, Sydney, Australia.
Pellom, Bryan, and Kadri Hacioglu. 2001. Sonic: e University of Colorado continuous speech recognizer. Technical report, Center for Spoken Language Research, University of Colorado, Boulder, USA.

190

BIBLIOGRAPHY

Pervouchine, Vladimir, Haizhou Li, and Bo Lin. 2009. Transliteration alignment. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL andthe 4th International Joint Conference on Natural Language Processing, 136–144, Suntec,
Singapore.
Pinker, Steven. 1994. e Language Instinct. New York, USA: Harper Perennial.
Ploux, Sabine, and Hyungsuk Ji. 2003. A Model for Matching Semantic Maps
between Languages (French / English, English / French). Computational Linguistics
29.155–178.
Ponsoda, Vicente, Julio Olea, Maria Soledad Rodriguez, and Javier Revuelta. 1999. e eﬀects of test diﬃculty manipulation in computerized adaptive
testing and self-adapted testing. Applied Measurement in Education 12.167–184.
Prichard, Caleb. 2008. Evaluating L2 readers’ vocabulary strategies and dictionary
use. Reading in a Foreign Language 20.216–231.
Pulido, Diana, and David Z. Hambrick. 2008. e virtuous circle: Modelingindividual diﬀerences in L2 reading and vocabulary. Reading in a Foreign Language
20.164–190.
Reips, Ulf-Dietrich. 2002. Standards for internet-based experimenting. Experimental
Psychology 49.243–256.
Rich, Elaine. 1983. Users are individuals: individualising user models. International
Journal of Man-Machine Studies 18.199–214.
Roget, Peter Mark, and Robert L. Chapman. 1977. Roget’s international thesaurus. New York, USA: Harper & Row.
Rosch, Eleanor, Carolyn B. Mervis, Wayne D. Gray, David M. Johnson,
and Peny Boyes-Braem. 1976. Basic Objects in Natural Categories. Cognitive
Psychology 8.382–439.
Rosen, Eric. 2003. Systematic irregularity in Japanese rendaku: How the grammar
mediates patterned lexical exceptions. Canadian Journal of Linguistics 48.1–37.
Saito, Hirafumi, Hisashi Masuda, and Masahiro Kawakami. 1998. Form andsound similarity eﬀects in kanji recognition. Reading and Writing 10.323–357.
Salton, Gerard, and Christopher Buckley. 1988. Term-weighting approaches
in automatic text retrieval. Information Processing and Management 24.513–523.

BIBLIOGRAPHY

191

Sato, Satoshi, Suguru Matsuyoshi, and Yoshsuke Kondoh. 2008. Automatic
assessment of Japanese text readability based on a textbook corpus. In Proceedings of
the 6th International Conference on Language Resources and Evaluation, Marrakech, Morocco.
Sears, Richard, 2008. Chinese Etymology Home Page. Available at http://www.
chineseetymology.org/. Accessed 10th Dec 2008.
Shannon, Claude Edward, and Warren Weaver. 1949. e Mathematical eoryof Communication. Urbana, USA: University of Illinois Press.
Shute, Valerie J., and Joseph Psotka. 1994. Intelligent Tutoring Systems: Past,
Present and Future. Technical report, Human Resources Directorate, Manpower andPersonnel Research Division, Brooks Air Force Base, Texas.
So, Sofumi. 2008. Review: Remembering the Kanji 1. e Modern Language Journal
92.663–664.
Taft, Marcus. 1991. Reading and the Mental Lexicon. London, UK: Lawrence Erlbaum
Associates.
——, Ying Liu, and Xiaoping Zhu. 1999a. Morphemic processing in reading Chinese.
In Reading Chinese Script: A Cognitive Analysis, ed. by Jian Wang, Albrecht W. Inhoﬀ,
and Hsuan-Chih Chen, 91–113. Mahwah, USA: Lawrence Erlbaum Associates.
——, Xiaoping Zhu, and Danling Peng. 1999b. Positional speciﬁcity of radicals in
Chinese character recognition. Journal of Memory and Language 40.498–519.
Taghva, Kazem, Julie Borsack, and Allen Condit. 1994. An expert system
for automatically correcting OCR output. In Proceedings of the IS&T/SPIE 1994 International Symposium on Electronic Imaging: Science and Technology, 270–278, San Jose,
California, USA.
Tanaka-Ishii, Kumiko, and Julian Godon. 2006. Kansuke: A kanji look-up system based on a few stroke prototype. In Proceedings of 21st International Conference on
Computer Processing of Oriental Languages, Sentosa, Singapore.
Toyoda, Etsuko, and Yoji Hashimoto. 2002. Improving a placement test battery: What can test analysis programs tell us. ASAA e-journal of Asian Linguistics andLanguage Teaching 2.
Tozcu, Anjel, and James Coady. 2004. Successful learning of frequent vocabulary through CALL also beneﬁts reading comprehension and speed. Computer Assisted
Language Learning 17.473–495.

192

BIBLIOGRAPHY

Tsujimura, Natsuko (ed.) 1999. e Handbook of Japanese Linguistics. Great Britain:
Blackwell Publishers Ltd.
Urquhart, Sandy, and Cyril Weir. 1998. Reading in a Second Language: Process,Product and Practice. New York, USA: Longman.
Vance, Timothy J. 1987. An Introduction to Japanese Phonology. New York, USA:SUNY Press.
Vermeer, Anne. 2001. Breadth and depth of vocabulary in relation to L1/L2 acquisition
and frequency of input. Applied Psycholinguistics 22.217–234.
Wagner, Robert A., and Michael J. Fischer. 1974. e string-to-string correction problem. Journal of the ACM 21.168–173.
Walters, JoDee. 2006. Methods of teaching inferring meaning from context. RELC
Journal 37.176–190.
Walz, Joel. 1990. e dictionary as a secondary source in language learning. e French
Review 64.79–94.
Wang, Liwei, Yan Zhang, and Jufu Feng. 2005. On the euclidean distance ofimages. IEEE Transactions on Pattern Analysis and Machine Intelligence 27.1–6.
Wills, Sebastian A., and David J. C. MacKay. 2006. DASHER – an eﬃcient
writing system for brain-computer interfaces? IEEE Transactions on Neural Systems andRehabilitation Engineering 14.244–246.
Winstead, Chris. 2006. Electronic kanji dictionary based on “Dasher”. In Proceedings ofthe 2006 IEEE Mountain Workshop on Adaptive and Learning Systems, 144–148, Logan,Utah, USA.
——, and Atsuko Neely. 2007. Kanjiru: a software interface for visual kanji search. In
Proceedings of the Forth International Conference On Computer Assisted Systems For Teaching & Learning Japanese, Honolulu, USA.
Yamamoto, Ken, and Yoshihiro Yamazaki. 2009. Network of two-chinesecharacter compound words in Japanese languages. Physica A: Statistical Mechanics andits Applications 388.2555–2560.
Yamashita, Hiroko, and Yukiko Maru. 2000. Compositional features of kanji foreﬀective instruction. Journal of the Association of Teachers of Japanese 34.159–178.
Yang, Tai-Ning, and Sheng-De Wang. 2001. A rotation invariant printed Chinesecharacter recognition systems. Pattern Recogntion Letters 22.85–95.

BIBLIOGRAPHY

193

Yeh, Su-Ling, and Jing-Ling Li. 2002. Role of structure and component in judgmentsof visual similarity of Chinese characters. Journal of Experimental Psychology: Human
Perception and Performance 28.933–947.
Yencken, Lars, Zhihui Jin, and Kumiko Tanaka-Ishii. 2007. Pinyomi - dictionary lookup via orthographic associations. In Proceedings of the 10th Conference for the
Paciﬁc Association of Computational Linguists, Melbourne, Australia.
Zhang, Dengsheng, and Guojun Lu. 2003. Evaluation of similarity measurement
for image retrieval. In Proceedings of the 2003 IEEE International Conference on Neural
Networks and Signal Processing, volume 2, 928–931.
Zhang, Jianna Jian, Howard J. Hamilton, and Nick J. Cercone. 1999. Learning English grapheme segmentation using the iterated version space algorithms. In
Proceedings of the 11th International Symposium on Methodologies for Intelligent Systems,
Warsaw, Poland.
Zock, Michael. 2002. Sorry, what was your name again, or how to overcome the tipof-the tongue problem with the help of a computer? In Proceedings of the SemaNet 2002
Workshop on Building and Using Semantic Networks, Taipei, Taiwan.
——, and Stergos D. Afantenos. 2007. Let’s get the student into the driver’s seat. In
Proceedings of the 7th International Symposium on Natural Language Processing, Chonburi,ailand.
——, and Slaven Bilac. 2004. Word lookup on the basis of associations: from an
idea to a roadmap. In Proceedings of the 20th International Conference on Computational
Linguistics, 89–95, Geneva, Switzerland.
——, and Julien Quint. 2004. Why have them work for peanuts, when it is so easy toprovide reward? motivations for converting a dictionary into a drill tutor. In Proceedingsof the 5th Workshop on Multilingual Lexical Databases, Grenoble, France.

Index
beginner’s paradox, 23

topology, 50

CATSS, 60

Heisig method, 30

classical test theory, 59

hiragana, 10

companion activation model, 42

homography, 112

distance metrics

inferring from context, 31

and alphabetic orthographies, 63

input method editor (IME), 14

bag of radicals, 66, 77

interactive-activation model, 45

L1 norm, 67

item response theory, 59

stroke edit distance, 79Japanesetree edit distance, 80

word formation, 18

within metric spaces, 65Japanese Language Proﬁciency Test
EDICT, 94, 99

criticisms, 136

ENAMDICT, 112

overview, 131
question types, 133

ﬂashcards, 38, 83, 129

JEDict, 55

Forgiving Online Kanji Search

JGloss, 39

as baseline, 92
background, 56

Kakasi, 113

limitations, 96

kanji

log analysis, 120

comparison to hanzi, 14

grapheme gapping, 20

compositional nature, 11grapheme-phoneme alignment, 95, 98

etymology, 11graphemic neighbours

homophony, 14

accessibility, 48, 49

pronunciation, 13
194

IndexKANJIDIC, 99

195
testingKanjiru, 17

adaptive, 59

Kansuke, 55

computer-based, 59

katakana, 10

vocabulary, 60

lexical gridding, 39
lexical relationships, 46, 177
mental lexicon, 41
mnemonics, 30
okurigana, 104
onbin, see sound euphony

tf-idf, 99, 102
tip of the tongue, 52, 97
transliteration, 112, 180
user model
granularity, 144
update algorithm, 150
validity, 58

orkurigana, 19
word
place names, 112

aspects of knowledge, 26

PopJisyo, 39

deﬁnition of, 22

Quizlet, 36

visual recognition, 42
WordNet, 46

Reading Tutor, 39
reliability, 58
rendaku, see sequential voicing
Reviewing e Kanji, 38
Rikai, 39
sequential voicing, 19
SKAT, 62
SKIP search, 53
sound euphony, 20
spaced repetition, 38
strategies
acquisition, 30
ordering, 28