Selecting Level-Specific BNC Applied Science Vocabulary Using Statistical Measures

Selecting Level-Specific BNC Applied Science
Vocabulary Using Statistical Measures
Kiyomi Chujo*
Masao Utiyama**
Nihon University*
National Institute of Information and Communications Technology**
chujo@cit.nihon-u.ac.jp
mutiyama@nict.go.jp
The effectiveness of using statistical measures to extract multi-level applied science
vocabulary from a large corpus was examined for the potential of being an easy-to-use tool
for teachers and developers of pedagogical materials. Nine statistical measures were applied
to a 7.37-million-word written ‘applied science’ component of the British National Corpus(BNC) to identify its domain-specific words, and an examination of the resulting vocabulary
lists showed 1) that each statistical measure extracted a different level of domain-specificwords by vocabulary level, grade level, and school textbook vocabulary coverage; 2) thatspecific measures produced level-specific words, i.e. beginning-level words were identified
by cosine and complimentary similarity measures, intermediate-level words were producedby the log-likelihood ratio, the chi-square test, and the chi-square test with Yates correction,
and advanced-level word lists were created with mutual information and McNemar’s test.
The authors conclude that the application of statistical measures can be an effective tool for
identifying and selecting level-specific BNC applied science vocabulary for pedagogical
purposes.

BACKGROUND
Because English is increasingly becoming a lingua franca for international technology andcommunications, there has been a growing interest in and necessity for English for Science andTechnology (EST). According to “the tree of ELT” (English language teaching — Hutchinson
and Waters, 1987), EST is categorized as one of the branches of English used for specificpurposes (ESP), all of which are differentiated from general English. This kind of English isimportant not only in scientific and technological activities but also in universities, which,increasingly, find themselves responsible for providing EST-related English skills to an everexpanding population of science and technology students.
Generally speaking, English classes at science and technology colleges in Japan consist of
three types: English for General Purposes (EGP), EST, and, “semi-EST” courses. EGP courses
are taught to freshman and sophomore students and are designed to further the student’s
abilities in using English as a communicative tool. Standard EFL college level textbooks areused in these courses. Usually EST courses are taught to seniors and graduate students and aredesigned to inculcate students with an ability to read and write the scientifically andtechnologically oriented English that they are likely to encounter in their professional careers.
This goal has been greatly underscored by the rise of English as the recognized international
language of science and technology. Consequently, technical articles from professional journals
are used in lieu of a textbook. Semi-EST courses are taught to juniors and are meant to bridge
the gap between the types of English used in EGP and EST classes; or, in other words, tosupply a transition between EGP and EST classes.
In ESP, or EST, one characteristic of the linguistic knowledge needed to comprehend
specialized texts is the heavy load of corresponding specialized vocabulary or “technical words
that are recognizably specific to a particular topic, field, or discipline” (Nation 2001:198), forthese words convey the import of the subject knowledge. In a previous study, we measured thegraduations among vocabulary levels found within EGP, semi-EST, and EST materials used at195

Kiyomi Chujo & Masao Utiyama

a college of science and technology, as measured by the 13,994-word lemmatized “British
National Corpus High Frequency Word List” (BNC HFWL) as a criterion. (This comparison
procedure is detailed in Chujo and Genung, 2003.) The resulting study confirmed the existenceof a large gap in the vocabulary level between the EGP and the EST teaching materials. The
study also revealed that the texts used in the semi-EST classes have only a limited efficacy inbridging this big gap, but that when supplemented by a specialized EST vocabulary list, they
can be more helpful in doing so.
We hypothesized that since the gap in vocabulary between the EGP and the EST teachingmaterials is so large that selecting and supplementing several levels of specialized EST
vocabularies that would supplement semi-EST courses according to learners’ proficiency levels
in a graduated, step-by-step fashion could probably best improve the efficacy of the English.
We know from Sutarsyah et al. (1994:48) that selecting multi-level beginning, intermediate,
and advanced specialized vocabularies using the traditional vocabulary selection criteria of
frequency and range is only partly successful in identifying the technical words. Because thefocus of these measures is ranking general-purpose vocabulary in order of priority, separating
technical vocabulary from general-purpose vocabulary is still labor-intensive, time-consuming,and heavily dependent on the selector’s expertise in English education and specialist
knowledge of the domain, which English teachers generally do not have. A means is clearlyneeded that provides easy-to-use tools that identify multi-level applied science vocabulary.
A number of corpus-based studies have used certain statistical measures to identify
technical vocabulary. For example, Nelson (2000) used the log-likelihood (LL) statistic fromWordSmith to find words that are statistically more frequently used in business English than ingeneral English by comparing each word’s frequency in the business English corpus with its
frequency in the British National Corpus (BNC). He was able to generate a list of businessrelated words such as business, market, customer, management, price, and bank.
In a preliminary study, Chujo and Utiyama (2004) examined a range of statisticalmeasures used in computational linguistics to identify technical vocabulary from a 100,000word specialized corpus. Eight measures such as the LL and the mutual information (MI) were
examined. They are statistics that indicate whether a word is overused or underused in a
specialized corpus compared with a corpus of general English. Each resulting list wascompared to an existing technical vocabulary control list, and the corresponding statisticalmeasures were evaluated for their effectiveness by calculating the proportion of relevant
candidates they produced. It was determined that all these measures effectively produce
relevant technical vocabulary and that each measure creates a unique type of word list that canbe specifically applied to student proficiency levels and lexicons. Our present study applies thesame methods but to a much larger corpus, includes an additional statistical measure, andexplores pedagogical applications based on BNC frequency, native speaker grade level, andJapanese textbook coverage.
Because of the lack of general agreement on how to define technical vocabulary (Justeson
and Katz, 1995), we must clarify some terms. The rank-ordered lists produced by eachstatistical measure are called specialized words/lists/vocabulary. We have used the broad
definitions of technical vocabulary; i.e., specialized lists contain three types of words:
technical vocabulary, or words specific to a field, sub-technical vocabulary, or words morecommon in the specified field than elsewhere and less obviously technical vocabulary, andgeneral vocabulary, which is a general base of English words.
RESEARCH QUESTIONS
We examined the effectiveness of using nine statistical measures to identify multi-levelapplied science vocabulary as easy-to-use tools for teachers and material writers. Specifically,
the following questions were addressed:

196

Selecting Level-Specific BNC Applied Science Vocabulary Using Statistical Measures

1. What types of applied science words are extracted by each measure, and how are theyranked?
2. How frequently do the top (most prominently appearing) 500 words extracted by eachmethod occur in the BNC?
3. At what U.S. grade level are the top 500 words extracted by each method understood?
4. What percentage of the top 500 words extracted by each method appear in Japanesehigh school (JSH) textbooks?
METHOD
Applied Science Master Word List
To extract applied science sub-lists, we needed to begin with one large master list of
applied science terms. To create this kind of applied science-related master list, we began withthe ‘applied science’ written component of the BNC. The 7.37-million words in this corpus
were first lemmatized to extract all base forms using the CLAWS7 tag set. Then, forpedagogical application, all unusual or infrequent words were eliminated by deleting words
which appear fewer than 100 times in the corpus. This created a list of 6,718 different words.
All proper nouns and numerals were identified by their part of speech tags and deleted
manually. Finally, this process yielded a 3,407-word applied science master list.
Control Lists
Three control vocabulary lists were used:(1) The British National Corpus High Frequency Word List (BNC HFWL), a list of
13,994 lemmatized words representing 86 million BNC words that occur 100 times or more(compiling procedure is detailed in Chujo, 2004), was used for comparison to statisticallydetermine if and how these applied-science-related words appear differently in a general corpus.
(2) The Living Word Vocabulary (Dale and O’Rourke, 1981) includes more than 44,000
items, and each has a percentage score that rates whether the word is familiar to students in U.S.
grade levels 4 through 16. This list was used to determine the grade level at which the central
meaning of a word can be readily understood.
(3) The junior and senior high school (JSH) textbook vocabulary list containing 3,245
different base words was compiled from the top selling series of JSH textbooks (the New
Horizon 1, 2, 3 series and the Unicorn I, II and Reading series) in Japan. Japanese high school
students generally use these or similar books to study English before entering a university.
Statistical Measures
The measures examined were mutual information (MI) (Church and Hanks, 1989), thelog-likelihood ratio (LL) (Dunning, 1993), the chi-square test (Chi2) and chi-square test with
Yates’s correction (Yates) (Hisamitsu and Niwa, 2001), the Dice coefficient (Dice) (Manningand Schütze, 1999), Cosine (Cosine) (Manning and Schütze, 1999), the complementary
similarity measure (CSM) (Wakaki and Hagita, 1996), McNemar’s test (McNemar) (Rayner
and Best, 2001) and frequency (Freq). These statistics automatically identify prominent words
by making comparisons between one specified list and another larger list. The formula for eachmeasure and a detailed description of each measure can be found in Utiyama et al. (2004). The
statistical score for the extent of each word’s “outstanding-ness” (Scott, 1999) in frequency ofoccurrence is computed, and the words are sorted from the most outstanding to the least
outstanding. Thus the words near the top are ranked as outstandingly prominent in terms ofeach statistical measure’s criteria.

197

Kiyomi Chujo & Masao Utiyama

The goal of identifying specialized words by using these measures is to narrow down thenumber of candidates for the category of technical or sub-technical items, not to totally extract
these items. Simply deleting the poor candidates would be a much simpler task for teachers andmaterial writers than creating the entire list manually.
RESULTS AND DISCUSSION
1. What types of applied science words are extracted by each measure, and how are theyranked?
The top 50 words from each of the nine different measures in ascending order are shownin Table 1. Since the top 50 extractions made using Freq and Dice, Cosine and CSM, and Chi2
and Yates were almost the same, they are shown in the same column. The bottom two rows ofeach column show the average frequency score and average word length of the top 50 words
generated by each statistical measure.
Table 1. Top 50 Specialized Words in the BNC Applied Science Corpus Generated byNine Measures
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
A ve ra ge
F re q u e n c y
A ve ra ge
W o rd L e n g th

F r e q , D ic e
th e
be
of
and
to
a
in
it
have
th a tfo r
w i thth i sonbyw i llth e y
as
not
or
fro m
can
w h ic h
at
you
u se
we
but
he
s y s te m
m any
do
m uch
m ay
a ll
th e r e
if
I
a lso
year
ti m eo th e rm ake
so m enewp a t ie n t
sa y
o n lyw hen
in t oC o sin e , C S M
th e
be
of
in
a
to
and
use
sy ste mfor
th is
w it hbyp a ti e n torw il lw h ic huser
c o m p u te r
fro m
so ftw a rein fo rm a tio n
a ls om ay
com panycannewstu d yp ro g ra m
show
te c h n o lo g y
f il ea p p li c a t io nprocess
productin c lu d e
asc e llre su lt
num berm o d u leeach
g ro u p
p ro v id e
d ise a sem any
c o n tr o lv e rsio n
year
w a te rLLs y s te m
p a tie n t
so ftw a reuser
c o m p u te r
use
of
th e
m o d u lebe
f i lete c h n o l o g y
in f o r m a ti o n
a p p lic a tio nc e lld a ta b a sed is e a s ev e r s io n
p ro d u ct
n e tw o rk
d is kpackageg a stric
p ro g ram
a c idp ro ce ss
in t e r f a c es tu d yserv ere n v iro n m e n t
th i scom panysunli b r a r yerro rm a c h in e
w in d o w
tr e a tm e n t
re su ltc o n c e n tra tio ns c i e n t is t
p ro ce sso r
in p u te l e c tr o n ic
w o r k s t a ti o ninty p ea v a i la b le
a lso
code

C h i2 , Y a te s
p a t ie n t
sy ste ms o f tw a r euser
c o m p u te r
m o d u leuse
f il ete c h n o lo g y
d a ta b a sec e llg a strica p p l ic a t io nofin fo rm a tio nd ise a se
th e
d isk
v e rsio nn e tw o rk
bein te rfa c e
packagea c idse rv e r
product
p ro g ra me n v iro n m e n t
p ro ce ss
p ro ce sso r
w o rk sta tio n
b ile
sun
e rro ru lc e r
stu d yin p u t
c o l it i s
h a rd w a red ig ita l
lib ra ry
s c ie n ti s t
c o n c e n tr a t io ndose
e le c tro n ic
m a c h in eb io p sy
m a in fra m e
c o l o n ic
d e sk to pMIc l ie n t-s e r v e rb il i a r ys i l ic o n
la bh y p e r te x t
ductin t e r o p e r a b il i ty
endangerp ix e lh ig h -e n d
re se lle rs e m i c o n d u c to rk ey w o rdm o ti l it ys c le r o t h e r a p y
m e ta p la sia
m a n o m e try
adenom am ucosale n d o sc o p ic
c o l o n ic
il e a lg a strind is t e n s i o n
a n tru m
f il e n a m e
m o th e r b o a r dduodenal
endoscopyc h o l a n g i t isre flu x
p la s m in o g e n
c o lo n o sc o p y
id i o p a t h i c
m a la b so rp tio n
il e u mr e s e c t io no b j e c t-o r ie n te d
sh a re w arep e p sin o g e n
g a strico n li n e
c o l it i s
p e r c u ta n e o u s
lu m i n a l
c h o l e c y s te c t o m y
u lc e r a ti v e
c o l o r e c ta l
p a n c r e a ti ti s
p a n c r e a ti c

M cN em ar
adeno m a
a n tru m
m a la b so rp tio n
p o s t p r a n d ia l
id i o p a t h i c
h y p e r te x t
lu m in a l
v a l u e -a d d e dc o lo n o sc o p yd is t e n s i o n
p e r c u ta n e o u s
m a n o m e try
sc le ro th e ra p y
n e u r o p a th yc o n n e c t iv it yp r o l if e r a ti v eduodenum
sc a la b leh is t o lo g yre c o g n ize rc h o l e c y s te c t o m y
m a c h in e -r e a d a b l e
ra d io c a rb o n
se c re to ry
h a b i tu a ti o n
asy n c h ro n o u s
su p p le m e n ta tio n
il e u m
la m i n ap e rfu sio n
m o th e rb o a rdin t e r o p e r a b il i ty
a u to n o m ic
u lc e r a ti o n
m a x im a l
ra d io lo g ic a la n a ly z e r
c irrh o sis
endan gerin s i g n i aly m p hr e c tu m
p a rie ta lth e s a u r u s
pancreasm e ta p la sia
c y to p la sm ic
b y teg l u te nlu m e n

58279

48263

28659

24038

343

123

3 .2

4 .9

6 .5

6 .7

9 .0

9 .6

198

Selecting Level-Specific BNC Applied Science Vocabulary Using Statistical Measures

The specialized lists in Table 1 are very different from each other even though they wereextracted from the same data. We can see that the BNC applied science corpora includes
various areas such as engineering, medicine, and chemistry. The top 50 words identified by
Freq and Dice are general vocabulary that usually appears at the top of high frequency lists inboth small and large corpora. For Cosine and CSM, the top 50 extractions are some important‘basic EST words’ that are all high-frequency words and have particular technical uses inapplied science, for example system, application, module, cell, patient and disease. The LL,
Chi2, and Yates seem to be well-suited to identifying sub-technical words in applied sciencesuch as interface, server, network, digital, and code in computer science and also gastric, acid,ulcer, bile, colitis and biopsy in medical science, and they appear to bridge the gap between
general English and technical words. The MI and McNemar lists identify EST technical words
such as hypertext, pixel, and semiconductor related to computer science and technology, biliary,
antrum, duct, postprandial, motility, histology, and endoscopy in the medical field.
As we see from the data in the bottom two rows of Table 1, the average frequency scoreof each list, ranging from 58279 to 123, decreases from left to right or from Freq to McNemar.
Inversely the average word length of lists increases from left to right, ranging from 3.2 to 9.6.
As Takefuta et al. (1994) have shown, difficulty levels in words increase with increasing word
length. This suggests that each measure identified different difficulty levels of words as its
outstanding words and this supports the possibility that specific statistical measures can beused to target specific grade-level vocabulary.
2. How frequently do the top 500 words extracted by each method occur in the BNC?
The BNC represents present-day general vocabulary usage. We examined the frequency
distribution of the top 500 extracted words by using the BNC HFWL, which was divided into
14 1000-word frequency bands of the most frequent words. ‘BNC frequency bands 1000’
indicates ranks 1 to 1000, ‘frequency bands 2000’ indicates ranks 1001 to 2000, etc. The
percentages of the 500 words of each list that belong to each frequency band are shown in
Table 2; a blank space indicates that no words belonged to that band. Because the scores for
Freq and Dice were identical, and those of Chi2 and Yates were almost the same, only sevencolumns are shown.
Table 2. Frequency Distribution of Top 500 Extractions
BNC Frequency
Bands

1,000
2,000
3,000
4,000
5,000
6,000
7,000
8,000
9,000
10,00011,00012,00013,00013,994

Freq, Dice

CosineCSM

LL

Chi2, Yates

MI

McNemar

92.4
6.6
0.8
0.2

65.8
12.0
6.8
5.0
3.4
3.4
2.4
0.6
0.4

59.8
20.8
10.2
5.0
2.2
1.8
0.2

34.8
16.0
13.0
9.4
6.0
7.0
6.0
3.0
2.6
1.0
1.0
0.2

29.8
13.8
11.6
9.0
6.4
7.6
6.8
4.6
4.0
2.8
2.0
1.4

2.2
4.2
7.2
8.0
10.0
12.2
12.2
10.8
12.4
8.2
4.6
3.8
3.0
1.2

3.6
30.2
20.8
12.2
12.4
8.2
4.6
3.8
3.0
1.2

0.2

0.2
> 20%

10-20%

199

2-10%

Kiyomi Chujo & Masao Utiyama

Table 2 shows clear graduations of frequency levels. Looking across the table from Freq
to McNemar, the top 500 words belong to increasingly lower frequency bands. The majority ofthe top 500 words from Freq and Dice belong to the top 1000 BNC. About 80% of Cosine andCSM belong to the top 2000 BNC. About 80% of LL, Chi2, and Yates words belong to top 5000
BNC or top 6000 BNC. Uniquely, MI extracts words evenly from all the frequency bands of
BNC HFWL words and about 80% belong to the top 9000 BNC. Interestingly, McNemar
extracts words only from ranks of lower frequency than BNC rank 5001, and 80% are extracted
from ranks 5001 to 10000. The nine statistical measures clearly extract different outstanding
levels of applied science words.
3. At what U.S. grade level are the top 500 words extracted by each method understood?
To understand grade level definitions for these extracted words, we examined the top 500
extractions for word familiarity by native English speaking children. Using The Living Word
Vocabulary (Dale and O’Rourke, 1981), which is “an inventory of the written words known by
children and young people in grades 4, 6, 8, 10, 12, 13, and 16,” (p. vii) we determined at what
grade level the majority of native English speaking students would readily understand thecentral meaning of each word in the top 500 extractions produced by the nine statisticalmeasures. The results are shown in Figure 1. ‘N/A’ denotes the words not appearing in The
Living Word Vocabulary. Most of N/A words are technical words such as colonic, ulcerative,
and mucosal.
100%

N/A
80%

16
13

60%

12
10
40%

8
6

20%

4
0%
Freq

Dice

Cosine

CSM

LL

Chi2

Yates

MI

McNemarFigure 1. U.S. Grade Level Based on Word Familiarity
The grade at which 60% of extracted words are understood can be used to clarify thegrade level comparisons: 60% of the top 500 words from Freq, Dice and Cosine are understoodby 4th grade students, those of CSM are understood by 6th grade students, those of LL, Chi2
and Yates are known by 8th grade students, those of MI and McNemar by 13th grade students.
This confirms that each statistical measure identifies different grade levels of words.
4. What percentage of the top 500 words extracted by each method appear in Japanese
junior and senior high school textbooks?
For this study to be meaningful in an EFL context, we must compare the vocabulary of the
top 500 words to an EFL standard. We compared the extracted words to the vocabulary learned200

Selecting Level-Specific BNC Applied Science Vocabulary Using Statistical Measures

by Japanese students. JSH text coverage is one way to obtain an accurate estimate of the
vocabulary level of each extraction, which is crucial information to EFL learners. For EST
learners who want to acquire applied science vocabulary, the percent of words that were notcovered by the JSH textbook vocabulary, represented by the upper section of each bar in
Figure 2, may be important information. Figure 2 graphically illustrates that while only 11%of the Freq/Dice top 500 extractions are not covered in the JSH school textbooks, 32% of the
Cosine, 36% of the CSM, 58% of the LL, and 63% of the Chi2/Yates extractions are notcovered, and 94% of the MI and 97% of McNemar extractions are not covered in the JSH
school textbooks. Overall the bar graph covered by the JSH textbook is similar to the bar graphof U.S. 4th grade in Figure 1. The data in Figure 2 again show that the nine different statisticalmeasures extract words of quite different grade levels.

100%
11

11
32

80%

36
58

63

63

60%
94
89

97

89

40%
68

64
42

20%

37

37

0%
Freq

Dice

CosineCSM% Covered by JSH

LL

Chi2

Yates

6

3

MI

McNemar% Not Covered by JSH

Figure 2. Percentage of Top 500 Words Covered / Not Covered by the JSH Textbook

CONCLUSION
In this study we applied nine statistical measures to the BNC applied science corpus andall of the data show that the statistical measures we used tend to extract specialized vocabulary
belonging to certain frequency bands and grade levels. Our results were similar to those ofprior studies based on a 100,000-word specialized corpus (Chujo and Utiyama, 2004, andUtiyama et al., 2004). Our study in combination with these previous studies shows that theresults of the statistical measures on corpora are quite similar even if the examined corpora
sizes are different.
The obvious pedagogical implication is that these statistical tools can very effectively be
used to extract various types of specialized lists that can be quickly and accurately targeted tolearners’ proficiency levels. For example, we can infer that the basic applied science words
extracted by Cosine and CSM would be good for beginning-level scientific English learners,
the LL/Chi2/Yates lists would be suitable for intermediate-level scientific English learners, MI
and McNemar would be appropriate for advanced-level scientific English learners, and Freq
and Dice might be useful for applied science students who need to consolidate JSH vocabulary
while learning basic scientific words. Thus we can use multi-level applied science lists thatwould supplement and bridge the big gap between EGP and EST in a graduated, step-by-stepfashion.

201

Kiyomi Chujo & Masao Utiyama

These statistical tools can help teachers and material writers to select sub-technical or
technical vocabulary without much specialist knowledge. Using extracted lists, they can easily
manually delete less relevant candidates. Further research will include developing these
extracted specialized lists into e-learning materials for vocabulary building.

REFERENCES
Chujo, K. and Genung, M. (2003). Vocabulary-level assessment for ESP texts used in the fieldof industrial technology. English Teaching, 58(3), 259–274.
Chujo, K. (2004). Measuring vocabulary levels of English textbooks and tests using a BNC
lemmatised high frequency word list. In: J. Nakamura, N. Inoue, and T. Tabata (Eds.),
English corpora under Japanese eyes (pp. 231–249). Amsterdam: Rodopi.
Chujo, K. and Utiyama, M. (2004). Toukeiteki shihyou wo shiyoushita tokuchougo chuushutsu
ni kannsuru kenkyuu (Using statistical measures to extract specialized vocabulary from a
corpus). KATE Bulletin, 18, 99–108.
Church, K. W. and Hanks, P. (1989). Word association norms, mutual information, andlexicography. Proceedings of ACL-89, 76–83.
Dale, E. and O’Rourke, J. (1981). The living word vocabulary. Chicago: World BookChildcraft International, Inc.
Dunning, T. E. (1993). Accurate methods for the statistics of surprise and coincidence.
Computational Linguistics, 19(1), 61–74.
Hisamitsu, T. and Niwa, Y. (2001). Topic-word selection based on combinatorial probability.
NLPRS-2001, 289–296.
Hutchinson, T. and Waters, A. (1987). English for specific purposes. Cambridge: Cambridge
University Press.
Justeson, J. and Katz, S. M. (1995). Technical terminology: Some linguistic properties and analgorithm for identification in text. Natural Language Engineering, 1, 9–27.
Manning, C. D. and Schütze, H. (1999). Foundations of statistical natural language processing.
Cambridge: The MIT Press.
Nation, I. S. P., (2001). Learning vocabulary in another language. Cambridge: Cambridge
University Press.
Nelson, M., (2000). A corpus-based study of business English and business English teachingmaterials. Unpublished Ph.D. thesis, University of Manchester, Manchester, England.
Rayner, J. C. W. and Best, D. J. (2001). A contingency table approach to nonparametric testing.
New York: Chapman & Hall/CRC.
Scott, M. (1996/1999). WordSmith Tools [Computer software]. Oxford: Oxford University
Press.
Sutarsyah, C., Kennedy, G., and Nation, P. (1994). How useful is EAP vocabulary for ESP? A
corpus-based study. RELC Journal, 25, 34–50.
Takefuta, Y., Hasegawa, S., and Chujo, K. (1994). Goi list ‘Gendaieigo no Keyword’ no nin’chi
level niyoru kubun no datousei (Validity of cognitive level grading for Keyword System
5000). Working Papers in Language and Speech Science, 4, 53–63.
Utiyama, M., Chujo, K., Yamamoto, E. and Isahara, H. (2004). Eigokyouiku no tameno bunya
tokuchou tango no sentei shakudo no hikaku (A comparison of measures for extracting
domain-specific lexicons for English education). Journal of Natural Language Processing,11(3), 165–197.
Wakaki, M. and Hagita, N. (1996). Recognition of degraded machine-printed characters using acomplementary similarity measure and error-correction learning. IEICE Trans. Inf. & Syst.
E79-D, 5.
Wynne, M. (1996). A post-editor’s guide to CLAWS7 tagging. Retrieved August 31, 2005,
from http://www.comp.lancs.ac.uk/computing/users/eiamjw/claws/claws7.html.

202