Towards building a usable corpus collection for the ELTclassroom
Kiyomi Chujo, Masao Utiyama and Chikako Nishigaki
Nihon University, National Institute of Information and Communications
Technology, and Chiba University
Abstract
As tantalizing as the potential for corpus application is in second language acquisition, we
educators seem to stumble over how to make concordancing lines understandable forlearners. This study explores various criteria for evaluating text samples by difficulty levelin order to provide a collection of rated parallel English-Japanese corpus texts which
educators can use in ELT classrooms, and provides the rating tools and methodology sothat educators can evaluate their own classroom material. Data was collected from two
English-Japanese parallel corpora, and seven indices (readability scores, average word
length, Japanese school textbook vocabulary coverage, BNC text coverage, Japanese
vocabulary ratio, sentence length, and kanji character ratios) were applied to measure thelinguistic difficulty of both English and Japanese text samples. It was noted that most of
the texts in the collection were advanced level, and that there is a shortage of copyrightavailable e-text data at the beginner level. Nevertheless, this study identifies several
applicable indices, provides a rated collection of titles at varying levels of difficulty, andtakes corpus usage one step closer to its ideal application.

1.

Introduction

Technology is changing our world and provides us with new tools for learning.
Using computers to produce corpora and concordancing data provides us withexciting new possibilities in our daily language-learning environment. Although
recognised by educators as a potentially useful tool, corpus application has both
highly contested advantages and disadvantages. Few attempts have been made touse corpora directly in the classroom by foreign language teachers and learners inJapan other than students of linguistics because of the difficulty students haveunderstanding the concordance examples retrieved (Tono 2003). One English as a
Foreign Language (EFL) learner looking at Figure 1 might easily beoverwhelmed by not only the long list of examples, but by vocabulary too
advanced to be useful. Addressing this widely acknowledged barrier to corpus
application is the subject of this paper. How can we as educators simplify
concordancing lines to make them understandable, and therefore useful, tolearners? Aston’s advice (2001: 43) is to carefully select the corpus or subcorpus;
and Thomas (2002) has provided a discussion of post-concordance filtering
according to each word frequency; however, to date no one has developed an
objective, easy-to-use criterion for evaluating the linguistic difficulty of various

48

Kiyomi Chujo, Masao Utiyama and Chikako Nishigaki

texts; and little research, if any, has tried to specifically investigate in what way
or ways the corpora texts are difficult for foreign language learners.

Figure 1. Comply in a monolingual corpus
2.

Research goals

To have understandable concordancing lines, we must begin with anunderstandable corpus. As Aston (2001) points out, the texts chosen for the
corpus must be selected carefully at an appropriate level for the learner. Therefore
the goals of this study were twofold: (1) to identify effective, easy-to-use criteria
for evaluating the linguistic difficulty of various English and Japanese texts or
subcorpora; and (2), once identified, to measure the linguistic difficulty of thevarious English and Japanese texts. This was done by creating a large parallel
corpus, extracting text samples, and then measuring the text samples with several
indices. The end product of this research is a collection of level-defined
subcorpora to be used for direct classroom application as well as the tools foreducators to apply to their own texts. The criteria can be used for selecting
appropriate-level data-driven learning material and reading textbooks, and inassisting web-searchers in choosing appropriate-level webpages. Since the levelof Japanese in the parallel corpus data was also evaluated, it is hoped that thismaterial is useful for learners of Japanese as well as English learners.
3.

MethodTo create a main corpus, a large number of text samples were collected into one
Parallel Corpora Text Collection (hereafter ‘the Collection’). The text samples
were extracted from the following two parallel corpora: (1) the English-JapaneseTowards building a usable corpus collection for the ELT classroom

49

Translation Alignment Data,1 which has 84 narrative and expository texts or
subcorpora written originally in English (632,564 words) and translated into
Japanese (1,011,873 morphemes), and aligned manually; and (2) the JapaneseEnglish News Article Alignment Data,2 which comprises 180,000 sentence pairs
from The Yomiuri Shimbun (6.1 million Japanese morphemes) and The Daily
Yomiuri (4.9 million English words), automatically aligned. This resulting
Collection therefore has two types of texts: stories and documents, and newspaper
articles. The story/document division contains 63 titles, encompassing a wide
selection of texts including stories (e.g. Jack and the beanstalk, The black cat),reading material in content areas (e.g. The Darwinian hypothesis, The declarationof independence) and blogs (e.g. Freedom or copyright?). All titles included inthis Collection’s story/document division were readily available e-texts eitherwith granted reproduction and redistribution rights by the copyright holders, oralready in the public domain. Unfortunately, the availability of these titles islimited, most notably at the beginner level.
4.

Referential data

In order for this study to be meaningful in an EFL or JFL (Japanese as a Foreign
Language) context, we must compare the vocabulary of the Collection to a
standard.3 In this case, a comparison of the English text vocabulary was made to
the vocabulary learned by Japanese students by calculating text coverage with the
top selling series of junior and senior high school (hereafter JSH) textbooks inJapan from the 7th through 12th grades.4 This vocabulary, totalling 3,098 different
words, is representative of the vocabulary studied by most college students before
entering university. The high-frequency words from the British National Corpus(BNC) were also explored as criteria. The certified standards of the Japanese
Language Proficiency Test: Test Content Specifications (hereafter JLPT Test
Content Specifications, see Kokusai Kouryuu Kikin 2002) served as a reference
guide for the Japanese texts.
5.

Text samples

A total of 99 sets of both English and Japanese sample texts were extracted fromeach of the two corpora discussed above. From the first – story and document –corpus, 87 sets of both English samples (on average 2,076 words) and Japanesesamples (on average 2,981 morphemes) were randomly extracted from the 63
titles. When the original text was small, the entire text was selected as a sample.
When the whole text was larger than the capacity of the Japanese language
analysis programme,5 two sets of samples were randomly extracted. English andJapanese samples are shown in Figure 2. From the second – newspaper – corpus,
12 sets of both English samples (on average 2,294 words) and Japanese samples(on average 2,963 morphemes) were selected randomly.

50

Kiyomi Chujo, Masao Utiyama and Chikako Nishigaki

English text
Every afternoon, as they were coming from school, the children used to go and play in the
Giant’s garden. It was a large lovely garden, with soft green grass. Here and there over the grassstood beautiful flowers like stars, and there were twelve peach trees that in the spring time brokeout into delicate blossoms of pink and pearl, and in the autumn bore rich fruit. The birds sat on
the trees and sang so sweetly that the children used to stop their games in order to listen to them.
“How happy we are here!” they cried to each other. One day the Giant came back. He had been
to visit his friend the Cornish ogre, and had stayed with him for seven years. After the sevenyears were over he had said all that he had to say, for his conversation was limited, and hedetermined to return to his own castle. When he arrived he saw the children playing in thegarden.
from The Selfish Giant, Oscar Wilde

Japanese text
子どもたちは毎日、午後になって学校から帰ってくると、大男の庭に行って遊ぶのが常でした。そこは、柔らかい緑の草が生えた、広くて素敵な庭でした。草むらのあち
こちには、星に似た美しい花が立っておりました。その庭には十二本の桃の木があり、春になると薄桃色と真珠色の繊細な花があふれるように咲き、秋には豊かな果実が実ります。鳥たちは木々の上でたいそう甘い歌声を響かせるので、子どもたちは遊ぶのをやめて聞きいるのでした。「ここで遊ぶのはなんて楽しいんだろう！」と、くちぐちに声をあげました。ある日、大男が帰ってきました。彼はコーンウォールに住む鬼の友人を訪問し、そこで７年間いっしょに過ごしていました。７年が過ぎ、話した
いことは全部話したし、もう話題もなくなってきたので、自分の城に帰ろうと思った

Figure 2. Text samples
6.

Indices investigated

Before deciding on the specific indices to apply in this study, the educational
literature was examined to understand which indices had been applied to measurelinguistic difficulty, and of those, which might be the most useful for this study.
We identified seven: four for English texts, and three for Japanese texts. The
indices were applied using various computer programmes; in cases where thewhole text was too large for the software programme used to measure the index,
two sets of samples were extracted from one title, and the average score of these
two samples was used as the title’s representing difficulty score. For the English
texts, the indices included (1) readability scores, (2) average word length, (3) the
text coverage of JSH textbook vocabulary, and (4) text coverage from the BNC.
We chose indices which could be applied by using readily available software sothat teachers would be able to apply these indices to their own data without
having to develop programmes themselves.
For JFL learners, it is Japanese that is the focus for learning; therefore it
was also important to measure the text difficulty of the Japanese text samples.

Towards building a usable corpus collection for the ELT classroom

51

JLPT Test Content Specifications state that the Japanese text content itself cannot
be measured for difficulty, but that it is possible to measure indices forquantitatively controllable variables on text difficulty such as vocabulary,sentence length and the number of kanji characters (Kokusai Kouryuu Kikin
2002: 219). Therefore to evaluate the Japanese samples, the indices applied were(5) the Test Content Specifications Levels 1 and 2 vocabulary ratios, (6) sentence
length, and (7) the percentage of kanji (characters). Each step is outlined below.
7.

English readability

7.1

Readability formulas

The term ‘readability’ refers to the factors that affect understanding a text andtherefore success in reading. In the context of this study, readability particularly
includes the complexity of words and sentences in relation to the reading abilityof the reader. We use the term ‘reading grade level’ to indicate the reading gradeof a text that could be read and just understood by a student of that grade who hasaverage reading ability; for example, a score of 8.0 means that an average nativespeaking eighth grader would understand the text. Objective measures ofreadability are generally done either by comparing a text with a standard word list
or utilising calculations involving the sentence length and number of syllables.
For this study, we calculated the readability score by using Readability
Calculations software,6 which contains nine widely used formulas including theFlesch-Kincaid formula (Flesch 1974). In our preceding study, we applied theseformulas to more than 100 text samples of various genres to observe thedifference among the yielded scores by different formulas (Chujo et al. 2004). It
was noted that three formulas, e.g. the Flesch-Kincaid formula, the SMOG
formula (McLaughlin 1969) and the Fry Graph (Fry 1968), were the mostreliable, as demonstrated by the range of grades observed from the difference
between each formula’s maximum and minimum grade level, and also from theappropriate within-group variability by the standard deviation. In order to provide
more validity for the current study and in an attempt to calibrate some fixed
points on the scale of readability, the averages of those three formulas wereapplied to the samples and the results are used to express ‘readability’.7
7.2

Comparing Japanese and English readability scores

Since much of the work on readability formulas has been done in the US, theformulas give a numerical value representing an American grade level. In orderfor these to be relevant to Japan’s educational situation, the readability of arepresentative sample of each JSH textbook by grade level was also calculated toprovide comparable measures to the readability scores of the targeted samples.
The procedure was as follows:

52

Kiyomi Chujo, Masao Utiyama and Chikako Nishigaki(1)

From each of the junior high school textbook series Horizon 1, 2 and 3,
which corresponds to the US 7th (Japan: junior high 1st), 8th (junior high2nd) and 9th (junior high 3rd) grades respectively, two reading lessons
entitled ‘Let’s read’ were selected, giving a total of six samples.

(2)

In the Japanese senior high school textbook series Unicorn I, II and
Reading, Unicorn I corresponds to the US 10th grade (Japan: senior high1st), Unicorn II corresponds to the 11th (senior high 2nd), and Unicorn
Reading corresponds to the 12th (senior high 3rd). From each of the
textbooks, three lessons (Lessons 1, 5 and 10) were selected, giving a totalof nine samples.

(3)

The readability scores of these 15 samples from the Japanese JSH English
textbooks in terms of American grade levels were measured by using thesame readability measures (the average of the three readability formulas)
as a scale.

Figure 3 shows the reading grade levels of the JSH textbooks investigated.
The vertical bars on the graph indicate the reading grade levels predicted by theuse of the three readability formulas. For example, the readability of the Japanese
first year junior high school textbook (US 7th grade) was rated as a (US) 2.8
reading grade. In other words, the English contained in the Japanese first year
junior high textbook might be readable by a US second grader nearing the end of
the school year. The reader may recall that two samples at the junior high (JH)level and three samples at the senior high (SH) level were used; these are
averaged together in the graph below.
10

Reading grade
8.9

8
6.4
6
4

3.7

6.0

4.3

2.8

2
0

7 th
7th
(JH
1 st)
(JH 1st)

8 th
8th
(JH
2 nd)
(JH 2nd)

9 th
9th
(JH
3 rd)
(JH 3rd)

10 th
10th
(SH
1 st)
(SH 1st)

11th
11th
(SH
2 nd)
(SH 2nd)

12 th
12th
(SH
3 rd)
(SH 3rd)Figure 3. The reading grade levels of JSH textbooks predicted by readability
formulasTowards building a usable corpus collection for the ELT classroom

53

Looking at Figure 3, we can see that the graduation of reading grade levels
among each grade textbook appears almost as one might expect. The 7th grade JH
1st textbook ranks the lowest, followed by 8th and 9th grade texts, then the gradelevel rises sharply to the 10th, then decreases slightly at the 11th, and finallyreaching the highest level with the 12th (SH 3rd) textbook. This graph indicatesthat the readability transition occurring in textbooks from JH to SH is neithersmooth nor easy. A discussion of the validity of the readability in JapaneseEnglish textbooks is an interesting and necessary discussion but one which goesbeyond the scope of the present article; nevertheless, the data in Figure 3 does
provide a means to compare measures for the readability scores of the target
samples. For the purposes of this study, we created three categories of linguistic
difficulty based on these results: texts lower than the 5.9 grade level were termed‘Level I,’ those falling between the 6.0 and 8.9 grade levels are ‘Level II’, andthose higher than the 9.0 grade level are ‘Level III’.
7.3

Average word length

Using Writers’ Workbench Version 8.15 Style Statistics,8 we obtained
information on basic stylistic variables such as average word length, sentence
length, the number of simple and complex sentences, the percentage of to be
verbs compared to the total number of verbs, and the percentage of use of passive
voice. Of these variables, only average word length provided applicable
information for this project. We can speculate that the other indices did not work
well because the sample text sizes may have been too small. Also note that noneof the three readability formulas described above uses ‘average word length’ bycalculating the number of letters, so measuring average word length provides
separate and additional data. As an added advantage, this index is easy to use andintuitively easy to understand.
7.4

JSH text coverage

The next calculation was the extent to which the vocabulary in the JSH texts doesor does not cover the vocabulary used in each of the text samples.9 This
constitutes one way of obtaining an accurate estimate of the vocabulary level ofeach text, which is crucial information to EFL learners. The ‘percentage
coverage’ refers to the percentage of the text that the learner is assumed tounderstand.
There has been continuing interest in whether there is a language
knowledge threshold which marks the boundary between having and not having
sufficient language knowledge for successful language use (Nation 2001). The
current thinking in the field of vocabulary teaching and learning puts thethreshold of meaningful input at 95% (ibid), therefore, 95% coverage was chosen
as the target. Thus the percent level of each sample text vocabulary not covered

54

Kiyomi Chujo, Masao Utiyama and Chikako Nishigaki

by the JSH textbook should be less than 5% in order to be understood by EFL
learners who studied English through these texts.
7.5

BNC text coverageWith more than 100 million words, the BNC is considered to be one of the mostreliable corpus resources available, and reflects present day English usage forspeech and publications in the UK. From the BNC, Chujo (2004) created alemmatised BNC high frequency word list of 13,994 words representing
86,123,934 words in the BNC occurring 100 times or more. The words are ranked
in terms of how frequently they are used, or how common they are. In teaching
EFL learners to recognise spoken or written words, it is obviously important toteach them those words they are most likely to encounter. In this study, the first
1,000, 2,000, 3,000, 4,000, 5,000 most frequent words from the BNC were used
as a criterion. Calculations of the percentage of words in each sample text notcovered by the top 1,000, 2,000, 3,000, 4,000, 5,000 BNC words were obtained.
7.6

Japanese vocabulary ratio

The JLPT Test Content Specifications divides Japanese language proficiency
levels into four, with Level 4 (beginner) as the first attained and Level 1 as thelast (advanced). The type and size of vocabulary are specified as follows: 800
words for Level 4, 1,500 words for Level 3, 4,800 words for Level 2, and 7,800
words for Level 1. We used the Vocabulary Level Checker programme,10 which
first divides the input text sample into words using the Chasen Version 2.02
Japanese morphological analysis system (Matsumoto et al. 1997), thenautomatically compares all the words in the text with the words in the four levelsof the JLPT Test Content Specifications, and finally shows the number of words
at each level in a classification table. Kawamura (1999) demonstrated that theratio of the sum of Levels 3 and 4 vocabulary highly correlated with text
difficulty, i.e. if a text sample contained a large number of Levels 3 and 4 words,
it was easier to understand; and the higher the ratio, the more easily it wasunderstood. In this study the converse was noted; that is, in the ratio of the sum ofLevel 1 and Level 2 words to the total number of words in the text, calculated asthe ‘Japanese vocabulary ratio’, it was noted that the higher the ratio, the moredifficult the text was considered to be.
7.7Japanese sentence length

Japanese sentence length is considered to be another measure that reflects thelevel of difficulty of texts. Sentence length was quantified as the average numberof Japanese characters per sentence in a text, and was counted using the CL Tool
programme.11 The average sentence length was obtained by dividing the total

Towards building a usable corpus collection for the ELT classroom

55

number of characters in the text by the number of sentences. The averagesentence length is also one of the numerical standards of the JLPT Test Content
Specifications: Level 4 average sentence length falls between 20 to 25 characters,
Level 3 is 25-30 characters, Level 2 is 30-45 characters, and Level 1 contains 4065 characters.
7.8

Kanji ratio

Japanese texts consist of kanji, hiragana, katakana, and other characters such as
English letters and numerals. The ratio of kanji to the total number of charactersin a text was counted by using the CL Tool programme. As with vocabulary andsentence length, the ratio of kanji is also considered to be one of the measures thatreflect the level of difficulty of texts (Kokusai Kouryuu Kikin 2002). For
example, JLPT Test Content Specifications specifies that Level 4 texts contain
15-20% kanji, Level 3 20-25%, Level 2 25-35%, and Level 1 30-45%.
8.

Results and discussion

The linguistic difficulty of both English and Japanese text samples measured byeach of the indices is shown in Table 1. Each title is shown with the title number,the author’s name, a narrative (N: white) or expository (E: grey) category, andeach index score. In order to grasp the distribution of the linguistic difficulty levelgraphically, the titles were sorted according to the readability scores from thelowest to the highest. Also, the range for each index score was divided into threelevels and colour-coded with ascending difficulty as follows: ‘Level I’ (white),‘Level II’ (light grey), and ‘Level III’ (grey). As we see from Table 1, with the
exception of the BNC text coverage, each index provides sufficient criteria forclassifying texts into three levels according to difficulty level. Furthermore, we
see that there are twenty-six titles that are broadly classified as ‘Level I’ inreadability, average word length or JSH text coverage, indicating that at least
there are some available titles that might be used at the beginner level for
Japanese students of English. Of these twenty-six, sixteen are rated ‘Level I’ withall three indices. Since the JSH text coverage indicators are in almost all cases
except one (Number 14: The selfish giant) well below the 95% coverageguidelines, the use of Japanese translations in parallel corpora might be helpful
for Japanese junior high school students. A few of these may be useful to JFL
learners, although the Japanese parallel corpus data ranked as ‘Level II’ for manyof these titles. It is interesting to note the differences in why certain texts might bedifficult, as shown by the different index scores. For example, Edgar Allan Poe’s
The tell-tale heart (Number 8) scores as ‘Level I’ in terms of readability andaverage word length, but as ‘Level II’ for JSH text coverage. This tells us thateven though this text may be at a US fourth grade reading level, and therefore
potentially accessible to Japanese junior high school students, the vocabulary
currently taught in Japanese schools would not support this kind of text. Also

O. Henry
C. & M. LambO Henry
O Henry
O. Wilde
C. Doyle
O Henry
J.M. Barrie
C. Doyle
C. Doyle
Ouida

10 The Gift of the Magi

11 A Midsummer Night’s Dream (retold)

12 Hearts and Hands

13 After Twenty Years

14 The Selfish Giant

15 The Adventure of the Norwood Builder

16 A Harlem Tragedy

17 Peter Pan in Kensington Gardens

18 The Adventure of the Blue Carbuncle

19 A Scandal in Bohemia

20 A Dog of Flanders

M. Lamb

9 The Tempest

O Henry

6 The Last Leaf
Andersen

L. Carroll

5 What The Tortoise Said To Achilles

E.A. Poe

Grimm

4 Snow White and the Seven Dwarfs

8 The Tell-Tale Heart

O. Wilde

3 The Happy Prince

7 The Little Match GirlJ. Jacobs

2 The Story of the Three Little Pigs

AuthorJ. Jacobs

Title

1 Jack and the Beanstalk

No.

N

N

N

N

N

N

N

N

N

N

N

N

N

N

N

N

N

N

N

N

5.4

5.3

5.2

5.2

5.1

5.1

5.0

5.0

4.9

4.8

4.8

4.7

4.7

4.6

4.5

4.3

4.2

3.6

2.6

2.4

4.2

4.1

4.1

4.1

4.1

4.1

4.0

4.0

4.2

4.1

4.1

4.1

4.0

4.2

4.1

4.1

4.0

3.9

3.6

3.7

Type ofWord
text Readability
length(grade)(letters)

12.6

9.6

10.5

6.8

12.3

9.0

3.3

9.0

10.0

10.6

10.0

8.8

10.2

8.7

11.7

7.0

8.7

6.6

7.8

14.9

17.8

17.2

11.0

17.6

15.9

10.5

15.1

12.9

12.8

15.4

12.4

14.9

13.7

13.9

16.2

11.6

12.7

7.3

14.1

ratio (％)(％)
6.4

Levels 1 & 2
vocabulary

Reference

39.7

23.1

35.1

39.7

34.2

31.4

32.2

28.7

30.5

33.3

32.6

29.0

30.1

30.2

32.7

22.7

27.7

33.8

24.4

25.2

20.2

26.4

21.6

16.4

18.8

21.4

22.2

18.8

20.0

17.2

18.5

19.7

23.5

19.1

19.7

18.3

17.6

22.1

14.3

15.8

23.6

16.7

17.5

15.4

21.1

15.6

12.9

15.4

18.3

20.6

19.7

18.4

18.1

17.8

21.3

13.2

15.3

15.9

17.5

16.1

Sentence
Kanji
Not covered
lengthcharacterby BNC(characters) ratio (%) top 1,000 (%)

Japanese indicesNot covered
by JSH text

English indices

Table 1. Index evaluation and rating for the parallel corpora text collection
56
Kiyomi Chujo, Masao Utiyama and Chikako Nishigaki

R.M. Stallman

E.A. Poe37 The Black Cat

40 JFK’s Inaugural Address. 1/20/1961

B. Clinton

36 Clinton’s Inaugural Address

A. Bierce

E.A. Poe35 The Assignation

R.M. Stallman

P. Seebach

34 The Manager FAQ

39 Waiting for the Knock

A. Bierce

33 An Imperfect Conflagration

38 The Man and the Snake

C. Dickens

J. London

30 The Shadow and the FlashA. Bierce

C. & M. Lamb

29 Romeo and Juliet (retold)

32 A Horseman in the Sky

M. Lamb

28 As You Like It (retold)

31 A Tale about a Queer Client

A. Strindberg

25 The Fad of The Fisherman

C. Doyle

G.K. Chesterton

24 To Build a Fire

27 The Adventure of the Devil’s Foot

J. London

23 The Arrest of Arsene Lupin

26 In Midsummer Days

C. Doyle
M. Leblanc

22 The Adventure of Charles Augustus Milverton

J. Joyce

21 Dubliners

8.0

Narrative
Expository

N:
E:

8.0

7.9

7.9

7.7

7.7

7.4

7.3

7.2

6.9

6.5

6.3

6.0

6.0

5.9

5.9

5.7

5.7

5.6

5.5

E

E

N

N

E

N

E

N

N

N

N

N

N

N

N

N

N

N

N

N

4.3

4.5

4.5

4.4

4.4

4.6

4.5

4.3

4.4

4.3

4.4

4.3

4.2

4.3

4.2

4.2

4.2

4.3

4.1

4.1

Level II
Level III

Grey

Level I

24.8

21.6

20.7

15.9

24.6

16.8

17.8

16.4

20.3

18.7

17.6

15.4

14.4

17.2

13.3

16.7

16.9

15.9

16.0

11.1

Light Grey:White:

11.1

11.2

14.4

18.1

10.4

14.7

13.4

14.0

12.3

11.3

12.3

12.7

10.5

9.8

8.7

8.5

9.6

11.1

10.0

6.1

47.5

42.6

36.9

49.2

41.2

51.2

37.0

40.8

33.9

37.3

34.3

44.5

36.4

35.7

50.5

26.2

28.8

26.6

34.4

26.6

27.8

23.7

28.6

22.9

25.2

25.6

21.0

25.6

26.3

26.4

21.3

21.0

21.6

24.1

18.2

23.4

22.9

21.6

17.8

22.3

21.3

16.5

23.9

23.0

18.6

25.2

15.3

19.8

23.4

21.6

21.7

20.6

18.3

17.4

19.3

17.4

20.1

19.2

17.4

13.8

Towards building a usable corpus collection for the ELT classroom
57

E

T. Veblen
R.M. Stallman

60 Why you should use the GNU FDL

E.A. Poe56 The Fall of the House of Usher

59 The Beginning of Ownership

FSF

55 Free Software is More Reliable!

R.M. Stallman

B. Black

54 The Abolition of Work

T.H. Huxley

FSF

53 Some Confusing or Loaded Words …

58 The Darwinian Hypothesis

R.M. Stallman

Why “Free Software’’ is better than “Open52
Source’’57 Freedom-Or Copyright?

E

Hiroo Yamagata

E

E

E

E

N

E

E

E

E

E

S.S. Van DineE

51 The Pragmatist of Free Software

R.M. Stallman

48 Linux and GNU-GNU Project

50 Twenty rules for writing detective stories

G.W. Schlabach

47 A Sense of History: Some Components

E

E

E

E

E

R. Landor

46 Interview with Karl MarxE
E

49 A Modest Proposal for preventing the children ... J. Swift

R.M. Stallman
R.M. Stallman

43 The Manager FAQ

45 Boycott Amazon!

Peter Seebach

42 A Biographical Sketch of an Infant

44 RMS Lecture at KTH (Sweden). 1986

J.F. Kennedy
C. Darwin

41 Someone to watch over me

Table 1. (Cont.)

11.5

11.2

10.4

10.2

10.2

10.1

10.1

10.0

9.6

9.6

9.5

9.1

8.8

8.8

8.7

8.5

8.4

8.2

8.1

8.0

5.1

4.8

4.7

5.0

4.6

4.9

4.9

4.9

4.7

4.3

4.6

4.5

4.6

4.8

4.4

4.6

4.2

4.4

4.3

4.5

18.8

18.0

14.7

16.0

18.4

19.2

16.6

16.6

9.6

8.3

16.8

15.0

10.5

10.4

11.6

11.8

5.9

7.6

8.8

9.0

29.6

28.2

23.6

24.4

19.9

24.3

25.8

27.6

27.0

17.4

26.3

21.0

20.4

19.8

25.1

22.6

15.8

19.2

16.1

20.0

56.7

63.4

68.7

42.5

63.0

62.6

44.3

55.3

55.0

37.9

38.5

44.7

55.1

34.9

43.8

49.4

47.1

43.3

43.6

52.2

23.1

36.5

30.5

27.4

23.3

16.8

31.4

27.2

20.8

17.0

33.4

32.1

15.0

27.8

25.5

23.2

16.8

25.7

26.8

23.6

9.5

22.5

21.7

21.4

21.2

26.4

24.8

21.5

20.0

15.6

13.0

27.5

21.2

10.7

16.3

16.4

15.9

10.5

10.1

16.8

58
Kiyomi Chujo, Masao Utiyama and Chikako Nishigaki

15.1

Narrative
Expository

E

N:
E:

75 The Yomiuri Shimbun/The Daily Yomiuri #12

14.3

14.1

14.0

14.0

E

E

71 The Yomiuri Shimbun/The Daily Yomiuri #8

13.8

13.8

74 The Yomiuri Shimbun/The Daily Yomiuri #11

E

70 The Yomiuri Shimbun/The Daily Yomiuri #7

EE

69 The Yomiuri Shimbun/The Daily Yomiuri #6

13.7

E

E

68 The Yomiuri Shimbun/The Daily Yomiuri #5

13.6

13.5

73 The Yomiuri Shimbun/The Daily Yomiuri #10

E

13.5

13.0

12.6

12.3

11.6

72 The Yomiuri Shimbun/The Daily Yomiuri #9

E

E

65 The Instinct of Workmanship …

67 The Yomiuri Shimbun/The Daily Yomiuri #4

E

66 The Yomiuri Shimbun/The Daily Yomiuri #3

E

T. VeblenE
E

64 The Yomiuri Shimbun/The Daily Yomiuri #2

R. Descartes

62 Discourse on the Method of Rightly …

63 The Yomiuri Shimbun/The Daily Yomiuri #1

USA.

61 The Declaration of Independence

5.3

5.3

5.1

5.1

5.1

5.1

5.1

5.1

5.1

4.8

5.0

5.1

5.0

4.5

4.9

Level II
Level III

Grey

Level I

36.9

36.1

30.5

32.5

32.0

31.1

32.8

32.0

32.3

26.8

31.9

31.2

32.3

20.4

27.5

Light grey:

White:

16.0

16.1

16.1

15.5

17.3

16.8

18.1

16.5

17.3

17.0

15.3

15.4

16.5

11.3

17.6

47.2

39.8

53.2

53.4

46.9

52.8

49.5

56.7

53.2

58.1

55.9

53.0

51.6

69.5

61.7

42.2

44.3

46.1

45.1

45.3

44.3

45.4

45.3

46.1

35.8

44.5

45.2

46.2

23.3

34.2

21.1

22.7

20.9

20.6

21.4

20.5

22.0

20.6

21.2

21.3

19.7

21.2

21.8

17.7

23.1

Towards building a usable corpus collection for the ELT classroom
59

60

Kiyomi Chujo, Masao Utiyama and Chikako Nishigakiinteresting is that although former US president Bill Clinton’s inaugural address(Number 36) is rated as ‘Level II’ in readability, average word length and JSH
text coverage, the Japanese parallel text ranks as ‘Level III’ for both vocabulary
and kanji. As we expected, each index for the twelve newspaper text samples
shows scores distributed within a narrow range, indicating they are fairly uniform
in their difficulty level, and that almost all indices rank the newspaper texts in one
classification (‘Level III’). From this we can predict that these newspaper texts
might provide a stable corpus for advanced level learners.
Surprisingly, the calculations for text coverage for the top 1,000 to 5,000
BNC words produced inconsistent scores. Only the top 1,000 BNC scores are
displayed in Table 1 as a reference to illustrate what scores were produced, andhow they compared to the other indices.
The criteria for dividing these data into three levels by each index are
shown in Table 2. A detailed discussion on each index result follows.
Table 2. Overview of level definitions
EnglishJapanese

Readability

Word
lengthPercentage
Percentage Levels 1 &
Kanji
not
Sentence
not covered
2
character covered byby JSH
vocabulary length
BNC topratio
vocabulary
ratio
1,000

Lower thanLevel US grade 5.9(JPN JuniorI
HS Level)

Shorterthan
4.2
letters

Less than
9.9%

Less than
15.9%

Fewer
than
30.4
characters

Fewer
than
19.9%

Less than
16.9%

Between US
Betweengrades 6.0 &4.3 &Between
8.9
4.5
10 & 14.9%(JPN Seniorletters
HS Level)

Between
16 &21.9%

Between30.5 &
44.9
characters

Between20 &24.9%

Between
17.0 &
20.9%

More than
22%

Longerthan
45
characters

More
than
25%

More than
21%

Level

Level
II

Higher than
US grade 9.0
Level
(JPN
III
College
Level)

8.1

Longerthan
4.6
letters

More than15%

Reference

English readability findings

The English readability score is shown in the 5th column of Table 1. This data
was classified into three rankings, based on the US reading grade and thecorresponding Japanese English school textbook readability score: ‘Level I’(lower than the US 5.9 grade level, i.e. at the Japanese junior high school English
textbook level), ‘Level II’ (between the US 6.0 and 8.9 grade levels, i.e. at the

Towards building a usable corpus collection for the ELT classroom

61

Japanese senior high school textbook level), and ‘Level III’ (higher than the US
9.0 grade level, i.e. at the Japanese college level or beyond). The non-shaded(white) scores indicate that the texts are at the appropriate level for Japanese
junior high school students, and the light-grey scores are titles appropriate forsenior high school students. It was noted that 15 titles and all of the newspaper
texts were higher than the 9.0 grade level, or higher than the Japanese senior highschool graduate level. Out of these 15 titles, 14 were expository texts. Reading
comprehension research tells us texts can be defined as either narrative (a fiction
or non-fiction story) or expository (non-narrative, an explanation or source of
information). Generally, narrative texts are easier to comprehend than expository
texts, since… [n]arratives possess a well-documented, familiar structure [and] …
from a content perspective, narratives typically deal with information
about social or interpersonal relationships and everyday problem
solving, content about which both adults and children tend to know
quite a bit. (Cote 1998: 6)
It is not surprising then that we see in Table 1 that many of the readability scoresfor expository material (newspaper articles, academic papers, commentaries,
blogs and political speech texts) are higher than the 9th grade level, and beyond
the reach of Japanese senior high school students:
Compared to narratives, expository text structures are more variable.
… [A] common purpose of expository texts is informational.
Informational texts frequently present concepts and relations thatreaders do not already know. They require that readers understand agreater range of logical relations among pieces of information. (ibid)
Thus, understanding expository texts generally requires more knowledge fromreaders, and consequently, is generally considered difficult by educators.
8.2

Average word length findings

The average word length of each text sample is shown in the 6th column of Table
1. The average number of letters was calculated, yielding a range from 3.6 to 5.3.
Studies such as Chujo and Takefuta (1989), and Takefuta, Hasegawa and Chujo(1994) showed that the longer the word, the higher the level of difficulty. Word
length was also used as one of the variables to classify texts in Biber (1988). In
Table 2, based on Chujo and Takefuta (1989), words shorter than 4.2 letters were
defined as ‘Level I’, and those between 4.3 and 4.5 letters were defined as ‘Level
II’. Those longer than 4.6 letters were defined as ‘Level III’. As we might expect,most narratives were evaluated as ‘Level I’ and most expository texts were
defined as ‘Level III’ in terms of average word length.

62

Kiyomi Chujo, Masao Utiyama and Chikako Nishigaki

8.3

JSH text coverage findings

The percentage of vocabulary not covered by the JSH textbooks is shown in the
7th column. This JSH textbook vocabulary represents the vocabulary a learnerusually acquires before entering a university. Researchers such as Laufer (1992)
and Nation (2001) pointed out that learners would need 95% text coverage tounderstand the meaning of texts. It turns out that only one text, The selfish giant(Number 14), fulfilled this criterion with a score of 3.3%. From this we can easily
imagine that those Japanese learners who studied English solely through school
English textbooks would have difficulty reading the English concordance lines ofany of the titles in the Collection except those from this one book. This resultshows that not only is the validity of Japanese textbooks (and their vocabulary
selection) called into question, but that, in spite of gaps in vocabulary learning inJapanese schools, the use of parallel concordancing lines showing Japanesetranslations would be not only helpful to learners, but perhaps essential. Teachers
using the ‘Level I’ titles shown in white under the JSH column, for example,
might have greater success if students also used the Japanese concordancing lines
since much, but not 95%, of the vocabulary in these titles is covered in theJapanese textbooks. It is also important to note that while the 95% coverage is theideal, the ‘Level I’, ‘Level II’ and ‘Level III’ guidelines in the study were set
respectively as 9.9%, between 10.0% and 14.9%, and above 15%. These divisions
were created in five point intervals within the distribution of the range of JSH text
coverage obtained from the titles. From Table 1, we see that the ratio of unknown
words is greater in expository texts than in narrative texts, which follows similar
observations in the readability and word length indices.
8.4

BNC text coverage findings

The BNC top 1,000 text coverage is shown in the far most right column. Since
the BNC represents present day general vocabulary usage, we expected that the
BNC high frequency word lists would function as an appropriate tool formeasuring vocabulary levels of the various texts investigated in this study.
Contrary to our expectation, the BNC lists were found not to correlatesignificantly with the other indices. As you can see from Table 1, there is aninconsistent variation on text coverage both between and within the ‘Level I’,‘Level II’ and ‘Level III’ text samples, indicating that BNC coverage is not asimilar predictor of level compared to the other six indices. While we believe that
BNC text coverage is a stable index, it may be that the BNC’s rating differences
are a factor of genre or publication dates in a way that the other indices are not.
For example, the BNC would rate blogs as ‘Level I’ and novels as ‘Level II’ or‘Level III’. If we accept that the BNC is a corpus of present day usage, this makes
sense since blogs, a very modern invention, are generally written with informal
language (sometimes in the style of personal journals), and novels usually use
more formal or descriptive vocabulary.

Towards building a usable corpus collection for the ELT classroom

63

So while at first glance the BNC did not seem to provide much useful data
for this study in terms of being a similar index to the others used, the dataobtained does raise interesting questions with regard to text selection, i.e. the typeof text (spoken, informative or imaginative) and its degree of modernity as afactor in how accessible it will be to learners. These questions warrant further
study, but because BNC text coverage was not a clear and consistent indicator oflevel regarding these specific texts, for the purposes of this study it was excluded
from additional between-index calculations. Also note that, as with JSH
classification, there was some degree of arbitration in defining the ‘Level I’,‘Level II’ and ‘Level III’ categories. For the percentage of top 1,000 BNC text
coverage, these were chosen from the distribution obtained (9.5% to 27.5%):
those less than 16.9% were defined as ‘Level I,’ those between 17.0 % and 20.9%
as ‘Level II’, and those more than 21% as ‘Level III’.
8.5

Japanese vocabulary ratio findings

The ratio of Levels 1 and 2 vocabulary specified in the JLPT Test Content
Specifications is shown in the 8th column of Table 1. There is no publishedstandard to classify the ‘Japanese vocabulary ratio’, so these were set within the
distribution obtained (7.3%-36.9%) as follows: those less than 15.9% were
defined as ‘Level I’, those between 10.0% and 14.9% as ‘Level II’, and thosemore than 22.0% as ‘Level III’.
The ratio of this advanced level vocabulary in narrative texts is roughly
about 15%, and is approximately 23% for expository texts except for newspaper
samples. Not unexpectedly, newspaper articles are at the 32% level. Thisconfirms the belief that expository texts use more difficult vocabulary thannarrative texts. This is a valuable finding for JFL learners using corpora.
8.6

Japanese sentence length findings

Japanese sentence length is shown in the 9th column in Table 1. The texts can beclassified into three groups based on the JLPT Test Content Specifications. The
titles which contain an average sentence length of fewer than 30.4 characters aredefined as ‘Level I’, and correspond to JLPT Levels 3 and 4. Titles having asentence length between 30.5 and 44.9 characters are defined as ‘Level II’, andcorrespond to JLPT Level 2. Those titles with an average sentence length longer
than 45.4 characters are defined as ‘Level III’; these correspond to JLPT Level 1.
Measuring Japanese average sentence length also clearly shows that expository
texts use longer sentences. Many of the expository text sentences are longer than
50 characters per sentence.

64

Kiyomi Chujo, Masao Utiyama and Chikako Nishigaki

8.7

Kanji ratio findings

The kanji character ratio is shown in the 10th column in Table 1. Titles can bebroadly classified into three groups according to the kanji ratio criteria shown in
Table 2. Again based on the JLPT Test Content Specifications, titles having akanji ratio fewer than 19.9% (JLPT Level 4) are rated as ‘Level I’, between 20.0
and 24.9% (JLPT Level 3) are ‘Level II’, and more than 25.0% (JLPT Levels 1
and 2) are ‘Level III’.
Kanji creates a huge learning burden for JFL learners or even Japanese
whose kanji proficiency level is low. Unfortunately for these learners, this study
showed that half of the Japanese texts investigated contained JLPT Level 1 or 2
kanji. Not surprisingly, newspapers in particular use a substantial amount ofkanji: on average, 45% of the text.
8.8

Correlation among indices

All of the above observations of the indices provided valuable insight regarding
the linguistic difficulty levels. Each, with the exception of the BNC,12 was asufficient criterion for classifying the particular texts investigated in this study.
We acknowledge that there was some degree of arbitration in defining difficultylevels for the indices, so in order to understand the extent of the similarity
between indices regarding the Level I, II and III classifications, we next
calculated the correlation between indices.
Table 3 shows the results for English texts and Table 4 for Japanese texts,
with high correlations (greater than 0.75) shaded to enhance clarity. Averaged
correlation coefficients between indices were shown in each bottom row.
Table 3. Correlation between indices for English texts
Readability

Word length

JSH text coverage

Readability

―

0.95

0.76

Word length

0.95

―

0.82

JSH text coverage

0.76

0.82

―

Average

0.85

0.88

0.79

> 0.75

Towards building a usable corpus collection for the ELT classroom

65

Table 4. Correlation between indices for Japanese texts
Levels 1 & 2
vocabulary
ratio

Sentence length

Kanji character
ratio

Levels 1 & 2
vocabulary ratio
Sentence length
Kanji character ratio

―

0.64

0.86

0.64
0.86

―
0.47

0.47
―

Average

0.75

0.56

0.67

> 0.75All three indices for the English texts showed a high correlation to eachother, and this correlation provides some support for the degree of arbitration
used in the study to define levels of difficulty. We see the strongest correlation
between readability and word length (0.95). Using the averaged correlation
coefficient as a means of comparison, the word length index showed the highest
correlation with other two English indices (0.88); thus we can say that this index
is regarded to be the most effective index among three indices observed in thisstudy in classifying English text difficulty. However, all three indices for English
texts showed a high correlation to each other, indicating that all three indices are
effective. Of these, the word length index will probably be the most easy-to-use
criterion for teachers since it can be obtained in one step with readily available
software such as Writers’ Workbench. Teachers in Japan evaluating texts forjunior and senior high school students may also want to consider JSH percentages
since, while the indices can and do reasonably target level, the percentage ofcoverage (or lack thereof) is a useful guideline unless students have the use ofJapanese parallel text data to compensate for the lack of known vocabulary.
We see the strongest correlation between Japanese Levels 1 and 2 vocabulary ratios and kanji character ratios (0.86) for Japanese texts. This would
follow conventional wisdom that higher level vocabulary would be expressed inkanji. Looking at the averaged correlation coefficient as a means of comparison,
Japanese Levels 1 and 2 vocabulary ratios showed the highest correlation with theother two Japanese indices (0.75); thus this index is regarded to be the mostuseful index among three indices observed in this study in classifying Japanese
text difficulty. This ratio can be calculated easily by using Vocabulary Checker
software available on the web (see Kawamura 1999).
Next, in order to explore the possibility that the English texts’ difficultylevel correlates with its Japanese translation texts’ difficulty level, for, in fact,
they are saying the same thing, we calculated the correlation between pairs ofEnglish and Japanese indices in Table 5. The three Japanese indices are on the top
row and the three English indices are in the first column. In the far right column,

66

Kiyomi Chujo, Masao Utiyama and Chikako Nishigaki

the averaged correlation between an English index and each of the three Japaneseindices are shown. In the bottom row, the averaged correlation between a
Japanese index and each of the English indices are shown.
Table 5. Comparing parallel English and Japanese text difficulty by indices
Levels 1 & 2
Sentence Kanji character
Averagevocabulary ratio length
ratio
0,93
0,77
0,85
0,85
Readability
0,76
0,66
0,66
0,69
JSH text coverage
0,93
0,71
0,82
0,82
Word length
0,87

Average

0,71

0,78

> 0.75Overall, there is a clear correlation between the English and Japaneseindices. It was interesting that English readability had the highest correlation
average with all three indices for Japanese texts, followed by word length.
Japanese Levels 1 and 2 vocabulary had the highest correlation average with all
three indices for English texts, followed by kanji ratio. We might infer from thisthat it might be possible to gauge the difficulty level of Japanese translation texts
from the English counterpart’s readability, and vice versa. For example, when
English texts are expository, it is not surprising that the Japanese translations
would use advanced Japanese vocabulary, longer sentences and a larger numberof kanji.
9.

Conclusion

In this study, the linguistic difficulty of English and Japanese text samples takenfrom two parallel corpora were measured with seven indices. Six were shown tobe applicable, and reliability was demonstrated by correlations between indices.
A text collection was created and the titles have been listed with each of the sevenindices (readability, word length, JSH textbook vocabulary coverage, BNC text
coverage, kanji ratio, Japanese vocabulary level and Japanese sentence length) todefine specifically in what way they are difficult. It was found that the best andeasiest way to evaluate the level of difficulty for English texts is by using averageword length, although readability scores and JSH textbook coverage can also be
useful. For measuring the level of difficulty in Japanese texts, vocabulary was themost effective. Also, the level of difficulty for one language was generally areliable predictor for the difficulty of the parallel language texts, i.e. a ‘Level I’
English text generally had a ‘Level I’ Japanese translation, and a ‘Level III’
English text corresponded to a ‘Level III’ Japanese translation, and vice versa.
Not surprisingly, it was found that many expository texts and all the newspaper
articles were difficult in both languages. The BNC text coverage data did notTowards building a usable corpus collection for the ELT classroom

67

provide comparable results but did raise interesting questions about the type(spoken, informative or imaginative) and modernity of texts chosen for learners,
especially those at the beginner level.
There is an unfortunate shortage of copyright available e-text material at
the beginner level, and the readability comparisons between American andJapanese grades discussed in this study might point to American graded readers
as a potential source of corpus data. In addition to the necessity for finding andincluding more beginner level corpus data, it is also clear that the vocabularytaught in Japanese junior and senior high schools may need review. In this study,
only one ‘Level I’ title contains vocabulary understood by the average highschool graduate at a 95% coverage level. Perhaps it is time for the Japanese
Ministry of Education, Science and Culture to create scientific and modern
vocabulary guidelines based on recent work in corpus linguistics.
A logical extension for future study would be to add easier level reading
texts to the text collection, to compare each index with the subjective difficultylevel standards obtained from educators, and to quantify and measure how muchthe difficulty level of English texts will be reduced by the use of Japanesetranslations. In the meantime, the use of Japanese parallel concordancing lines
might be a reasonable tool in understanding English texts, and a case study on
this topic is the subject of our next research project.
Notes
1English texts were mainly collected from Project Gutenberg (at«http://promo.net/pg/») and the GNU Project (at «http://www.gnu.org/»),with Japanese translations from Project Sugita Genpaku (at «http://www.
genpaku.org/») and from other resources. The project (Utiyama 2003) ison-going and the corpus is available at «www2.nict.go.jp/jt/a132/
members/mutiyama/align/index.html».

2This corpus was created by Utiyama and Isahara (2003) and is available at«www2.nict.go.jp/jt/a132/members/mutiyama/jea/».

3

For a discussion on how the English texts were prepared for readability
and text coverage calculations, see Chujo et al. (2004).

4

The following textbook series were used: Asano et al. (1999) and Suenaga
et al. (2001).

5

Vocabulary Checker available at «http://language.tiu.ac.jp/tools.html» (seeKawamura 1999).

6

Micro Power & Light Co. 2003:«http://www.micropowerandlight.com».

7

These formulas use one or more of the following criteria to calculate thescore: number of words, number of syllables, number of sentences,

Readability

Calculations.

68

Kiyomi Chujo, Masao Utiyama and Chikako Nishigaki

average number of syllables, and number of words more than threesyllables.
8

EMO Solutions 2004: Writer’s«http://www.emo.com/wwb/».

Workbench

Version8.15

9

We used our own programme for calculating text coverage, but a similar
programme is available at Paul Nation’s web site «http://www.vuw.ac.nz/
lals/staff/paul-nation/nation.aspx».

10

It is worth noting that this software is only capable of handling 19KB ofdata (about 200 sentences), so two random samples of 200 sentences fromeach title were extracted and averaged together to calculate the score.

11

CL TOOL Version 1.2 «http://sano.tufs.ac.jp/cltool/» (see Sano 2003).

12

The correlation of the BNC top 1,000 percentages with readability, word
length and JSH text coverage was .37, .45, and .77 respectively. Its
correlation with Japanese vocabulary, sentence length and kanji ratio was.41, .32, .43 respectively.

References
Asano, H., Y. Shimomura, T. Makino, M. Ikeda, A. Ikeya and S. Ishizuya (1999),
New Horizon English Course 1, 2, and 3. Tokyo: Tokyo Shoseki.
Aston, G. (2001), Learning with corpora. Houston: Athelstan.
Biber, D. (1988), Variations across speech and writing. Cambridge: Cambridge
University Press.
Chujo, K. (2004), ‘Measuring vocabulary levels of English textbooks and tests
using a BNC lemmatised high frequency word list’, in: J. Nakamura, N.
Inoue and T. Tomoji (eds.), English corpora under Japanese eyes.
Amsterdam: Rodopi, 231-249.
Chujo, K. and Y. Takefuta (1989), ‘Joseimuke Eigo zasshi no goi (vocabulary of
women’s magazines)’, Current English Studies, 28: 73-84.
Chujo, K., A. Shirai, M. Utiyama, C. Nishigaki and S. Hasegawa (2004), ‘Nichiei
parallel corpus wo kouseisuru text no nan’ido ni kansuru kenkyuu (a study
on classifying texts in English-Japanese parallel corpora according tolinguistic difficulty)’, Journal of the College of Industrial Technology
Nihon University, 37: 57-68.
Cote, N., S.R. Goldman and E.U. Saul (1998), ‘Students making sense of
informational text: relations between processing and representation’,
Discourse Processes, 25: 1-53.
Flesch, R. (1974), The art of readable writing. New York: Harper and Row.
Fry, E. (1968), ‘A readability formula that saves time’, Journal of Reading, 11/7:
265-271.
Kawamura, Y. (1999), ‘Analysis of Japanese textbooks using the “Vocabulary
Level Checker”’, in: K. Nakajima et al. (eds.), Proceedings of the Second

Towards building a usable corpus collection for the ELT classroom

69

International Conference on Computer Assisted System for Teaching &Learning Japanese. Toronto: University of Toronto, 132-137 (retrieved
from «http://language.tiu.ac.jp/about.html»).
Kokusai Kouryuu Kikin (2002), Nihongo Nouryoku Shiken Shutsudai Kijun(Japanese Language Proficiency Test: Test Content Specifications)(Revised Edition), Tokyo: Bonjinsha.
Laufer, B. (1992), ‘How much lexis is necessary for reading comprehension?’, in:
L. Arnaud and H. Bejoint (eds.), Vocabulary and applied linguistics.
London: Macmillan, 126-132.
Matsumoto, Y., K. Kitauchi, T. Yamashita, O. Imaichi and T. Imamura (1997),‘Nihongo keitaiso system Chasen manual version 1.0’. NAIST Technical
Report 97007. (Version 2.3.3 is available at «http://chasen.aist-nara.ac.jp/
chasen/manual.html.ja») .
McLaughlin, G. (1969), ‘SMOG grading: a new readability formula’, Journal of
Reading, 12/8: 639-646.
Nation, P. (2001), Learning vocabulary in another language. Cambridge:
Cambridge University Press.
Sano, H. (2003), Windows PC niyoru Nihongo kenkyuuhou (methods ofinvestigating Japanese language by use of Windows PC). Tokyo: Kyoritsu
Shuppan Co. Ltd.
Suenaga, K., Y. Yamada, K. Fukai, S. Nakamura, K. Ishizuka and K. Ichinose(2001), Unicorn English Course I, II, and Reading. Tokyo: Bun’eido.
Takefuta, Y., S. Hasegawa and K. Chujo (1994), ‘Goi list “Gendaieigo no
Keyword” no nin’chi level niyoru kubun no datousei (validity of cognitive
level grading for Keyword System 5000)’, Working Papers in Language
and Speech Science, 4: 53-63.
Thomas, J. (2002), ‘Concordancing for all’, paper presented at the Fifth Teachingand Language Corpora Conference, Bertinoro, Italy, 27-31 July 2002.
Tono, Y. (2003), ‘Corpus wo Eigo kyoiku ni ikasu (what corpora can do for
language teaching)’, English Corpus Studies, 10: 249-264.
Utiyama, M. (2003), ‘Japanese-English bilingual corpora and their applications’,
demonstration presented at the Asialex Conference, Tokyo, Japan, 27-29
August 2003.
Utiyama, M. and H. Isahara (2003), ‘Nichiei shimbun no kiji oyobi bun wo
taiouzukeru tameno koushinraisei shakudo (reliable measures for aligning
Japanese-English news articles and sentences)’, Journal of Natural
Language Processing, 10/4: 201-220.