Poster Presentation

Harnessing Keyness: Corpus-based Approach to ESP MaterialDevelopmentJohn Blake
Japan Advanced Institute of Science and Technology
Concordancers often provide an option to generate lists of keywords. Keywords arethe words that occur disproportionately more frequently in a particular text type (e.g.,business English) compared to another text type (e.g., general English). This is one wayof distinguishing technical or domain-specific words from general words. Novice usersof concordancers tend to expect that the keyword lists produced are identical, yet there
are significant differences in the lists generated. This paper shows how keyword lists areaffected by the choices of concordancer, reference corpus and statistical test. ESP materials
developers can use this knowledge to make a more informed choice of the variables so thatthe most appropriate keyword list for the target audience can be created.

The identification of words that deserve inclusion in teaching materials is adifficulty that many materials developers face. There are many factors to consider
in the selection of vocabulary, such as frequency, appropriacy, expediency, needand level. The most frequent words in a text are relatively easy to identify, but
are not necessarily the most useful words to highlight in ESP materials. Thisis because grammatical words and high frequency general words are likely
to occupy the top positions. Words that are key, however, are likely to meritinclusion. Specialized software programs (concordancers) can analyze collectionsof written texts (corpora) to identify the frequency and keyness of vocabulary.
Simply put, keyness is a measure of the frequency with which a word occurs
in the corpus being analyzed (focus corpus) in comparison to another corpus(reference corpus). Words that occur more frequently show positive keyness and102

Harnessing Keyness, OnCUE Journal, 9(2), pages 102-110

are commonly called key words (Scott, 1997). For example, the words export,
firm and market are more likely to occur in texts about business than in texts
on general topics, so these words show positive keyness. Keyness is computed byusing a concordancer to count words in the focus corpus, compare the counted
frequency to counts in a reference corpus by using a statistical test. The choiceof concordancer, reference corpus and statistical test affects the keyword lists
generated.
Novice users may expect all concordancers to produce the same keyword
list for a text. However, this is not the case. Different concordancers, reference
corpora and statistical tests result in radically different keyword lists.
Concordancers can be classified into four generations (McEnery and Hardie,2012) although the first two generations are now obsolete. Fourth-generation
concordancers can deal with large corpora and are far more powerful than thirdgeneration concordancers, such as AntConc (Anthony, 2012) and Wordsmith
Tools (Scott, 2012) (Table 1). Some concordancers provide options to uploada reference corpus to which the focus corpus can be compared, while others
provide a range of corpora from which the user can select. Concordancers
may have a default statistical test (e.g., chi-squared in AntConc) or provide
alternatives for the user to select. Keyword list generation is underpinned bycomparing the ratios of words occurring in the focus and reference corpora
using statistical tests. Kilgarriff (2012) highlights two statistical problems whencomparing two corpora. The first is the resolution of dividing by zero when
Table 1
Current Generations of Concordancers
3rd generation

4th generation

Location

Personal computers

Web servers

Size of corpora

Small corpora - low millions

Large corpora – 100 million+

Examples

AntConc (Anthony, 2012)
CQPweb (Hardie, 2012)
UAM Corpus Tool (O’Donnell, 2013) Sketch Engine (Kilgariff et al.,Wordsmith Tools (Scott, 2012)2014)
W-matrix (Rayson, 2008)103

Blakethere are no occurrences of a word in the reference corpus. The second involves
overcoming the domination of words which occur rarely in the reference corpus.
Different tests use different methods to address these issues.
This paper explores how the choice of concordancer, reference corpus andstatistical test generates different lists of keywords. Materials developers can use
this knowledge to make more informed choices of which vocabulary to focus onin their tailor-made materials.

MethodA corpus of texts comprising all the research articles published in the journalInternational Business Review from February 2010 to October 2013 wasmanually collected and concatenated into a single text file. Table 2 shows thecomposition of this focus corpus.
The three variables (concordancers, reference corpora and statistical tests)
were each tested in turn. A popular third-generation concordancer, AntConc
3.2.4w (Anthony, 2012), and a popular fourth-generation concordancer, Sketch
Engine (Kilgarriff et al., 2014), were selected for comparison. The raw frequency
word count for each concordancer was first calculated. Keyword lists weregenerated using the British Academic Written English (BAWE) corpus and theBrown corpus in Sketch Engine (Table 3). A keyword list was then generatedusing the Brown corpus in AntConc. This was undertaken using three simple
maths statistical tests in Sketch Engine and two simple ratio tests in AntConc.
The keyword lists were then evaluated from the perspective of an ESP materials
developer.
Table 2IBR Focus Corpus
Count (made in AntConc 3.2.4w)
Tokens2,516,051

Words

1,966,650

Sentences

77,547104

Harnessing Keyness, OnCUE Journal, 9(2), pages 102-110

Results
Concordancers
The raw count of frequency of words in both AntConc and Sketch Engine
results in the same order for the top ten words, yet only the word count forthat is identical (Table 4). This raw word count difference can be accounted
for by differences in the operational definition of a word and the process oftokenization. For example, Anthony (2013) notes that Wordsmith Tools andAntConc count contractions differently, e.g., we’ll is counted as one word in
Wordsmith, but as two words in AntConc. Word count is just one variable inthe calculation of keyness. Since results differ at the level of raw word count, thisdifference may be exacerbated by the choice of reference corpus and statistical
test.
Each concordancer offers different functionality with regard to calculating
keyness. For example, AntConc allows users to upload their own reference corpus
and provides the standard choice of either chi-squared or log-likelihood for thestatistical test, while Sketch Engine incorporates access to numerous reference
corpora and a set of statistical tests based on simple maths (Kilgarriff, 2009).
For most ESP material developers, the functionality of the concordancer is most
likely of more importance than a thorough understanding of the definition ofwords and tokenization process used. Materials developers want to know which
vocabulary deserves inclusion in materials and so selecting a concordancer thatcan produce key word lists that are pitched at the level and topic of the target
audience is of primary importance.
Table 3Outline of Reference Corpora Used
BAWE corpus

Brown corpus

Date created

2000s

1960s

Type of corpus

AcademicGeneral

Type of EnglishBritishAmericanWords

6,506,995

1,000,000105

BlakeTable 4Raw Frequency Results
NoSketch EngineAntConc

1

the106,022

106,0642

and77,508

77,542

3

of72,73372,9904

to47,454

47,834

5

in41,79142,056

6

a32,00732,336

7

that

23,09223,0928

is21,249

21,2459

for17,293

17,30310

as14,30914,329Reference Corpora
Table 5 shows the keyword lists created in Sketch Engine using the Midway
statistical test but with different reference corpora. Keyword lists created whenusing the BAWE corpus and Brown corpus shared five of the top ten results.
The remaining five words in BAWE appeared more specialized than the Browncorpus. The BAWE keyword list, therefore, appears more appropriate for learners
with a stronger vocabulary base.
Scott (2009) claims that there is no bad reference corpus. However, different
reference corpora yield radically different keyword lists. Keyness is significantly
affected by the genre and diachrony of a reference corpus (Goh, 2010). Whenthe focus and reference corpus are more similar in terms of topic and time
period, the keyword list is likely to contain words that are more obscure. As theBAWE corpus is more similar in terms of genre and diachrony than the Browncorpus to the focus corpus, the resultant key word list therefore contains words
that are more obscure (Table 5). Given that different reference corpora impact
the generated keyword lists, ESP materials developers would be well advised tocompare the results using different reference corpora.
106

Harnessing Keyness, OnCUE Journal, 9(2), pages 102-110

Statistical Tests
As shown in Table 6, selecting the log-likelihood and chi-squared tests in
AntConc using the Brown corpus resulted in identical lists for the first eightkeywords. Simple ratios, such as log-likelihood and chi-squared, produce
keyword lists “dominated by rare words” (Kilgarriff, 2012, p.5). Both of thesetests are based on the assumption of randomness; but language is not random(Kilgarriff, 2005), and so these tests are inappropriate (Gabrielatos and Marchi,2012).
Table 7 shows the keyword lists generated in Sketch Engine using theBAWE Corpus, but selecting different statistical tests. The simple maths version(Kilgarriff, 2009) in Sketch Engine names the tests clearly (e.g., Common, Rare)and is not based on the assumption that language is random. Rare resulted inhigher occurrence of rare words, while Common resulted in a skew to morecommon words. When selecting vocabulary for less proficient students, it may
be prudent to use a keyword list generated using Common.

Table 5Keyword Lists using BAWE and Brown in Sketch Engine with Midway Test
NoBAWE

Brown

1

firms

firms

2

firm

firm

3

exportexport4

foreignTable

5subsidiaryvariables

6

internationalizationinternational

7

FDImarkets

8

subsidiaries

knowledge

9markets

foreign10MNEs

market107

BlakeTable 6
Keyword Lists using Log-likelihood and Chi-squared Tests in AntConc with Brown Corpus
No

Log-likelihood

Chi-squared

1thethe2

firms

firms

3

firm

firm

4

alet

5

etal

6

inIn

7

knowledgeknowledge

8marketmarket9

thisinternational

10

table

foreign

Table 7
Keyword Lists using Three Statistical Tests in Sketch Engine with BAWE Corpus
No

Rare

MidwayCommon

1

OFDI

firms

and

2

offshoring

firm

firms

3

Vahlneexportfirm

4

multinationality

foreign

foreign5

Full-size

subsidiaryknowledge6MathML

internationalizationinternational

7

Kogut

FDImarket

8BOP

subsidiaries

country

9

MathJax

markets

Table

10

Ghoshal

MNEs

performance108

Harnessing Keyness, OnCUE Journal, 9(2), pages 102-110

Conclusion

The three variables of concordancer, reference corpora and statistical tests greatlyaffect the keyword lists generated. The functionality of fourth-generation
concordancers far outweighs third-generation, and so if time is a priority, it isworth investing in a subscription. Sketch Engine provides an easy, quick, andaffordable way to calculate a variety of keyword lists. The availability of 20
reference corpora and four appropriately-named statistical tests make it easy fornovice users to tailor keyword lists to the intended learners. Selecting a general
English reference corpus and the Common statistical test in Sketch Engine islikely to generate keyword lists that are more suitable for lower level students.

References

Anthony, L. (2012). AntConc (Version 3.2.4) [Computer Software]. Tokyo,
Japan: Waseda University.
Anthony, L. (2013). A critical look at software tools in corpus linguistics.
Linguistic Research, 30(2), 141-161.
Gabrielatos, C., & Marchi, A. (2012). Keyness: Appropriate metrics andpractical issues. Paper presented at Corpus-assisted Discourse Studies
International Conference 2012, University of Bologna, Italy. Retrieved
from http://repository.edgehill.ac.uk/4196/
Goh, G-Y. ( 2010). Choosing a reference corpus for keyword extraction.
Linguistic Research, 28(1), 239-256.
Hardie, A. (2012). CQPweb—combining power, flexibility and usability in acorpus analysis tool. International Journal of Corpus Linguistics, 17(3), 380409.
Kilgarriff, A. (2005). Language is never ever ever random. Corpus Linguistics
and Linguistic Theory, 1(2), 263-276.
Kilgarriff, A. (2009). Simple maths for keywords. In M. Mahlberg, V. GonzálezDíaz & C. Smith (Eds.), Proceedings of the Corpus Linguistics Conference
CL2009. Retrieved from http://ucrel.lancs.ac.uk/publications/cl2009/
Kilgarriff, A. (2012, January). Getting to know your corpus. In International
Conference on Text, Speech and Dialogue (pp. 3-15). Berlin: Springer Verlag.
109

BlakeKilgarriff, A., Baisa, V., Bušta, J., Jakubíček, M., Kovář, V., Michelfeit, J., ...
Suchomel, V. (2014). The Sketch Engine: Ten years on. Lexicography, 1(1),7-36.
McEnery, T., & Hardie, A. (2012). Corpus linguistics: Method, theory andpractice. Cambridge: Cambridge University Press.
O’Donnell, M. (2013). UAM Corpus Tool (Versions 2.8 & 3.1) [ComputerSoftware]. Edinburgh, UK: Wagsoft Systems.
Rayson, P. (2008). W-matrix corpus analysis and comparison tool. Lancaster,
UK: Lancaster University. Retrieved from http://ucrel.lancs.ac.uk/
wmatrix/
Scott, M. (1997). PC analysis of key words – and key key words. System, 25(1),1-13.
Scott, M. (2009). In search of a bad reference corpus. In D. Archer (Ed.), What’s
in a word-list? Investigating word frequency and keyword extraction (pp.7992). Oxford: Ashgate Publishing.
Scott, M. (2012). WordSmith Tools (Version 6) [Computer Software].
Liverpool: Lexical Analysis Software.

Author bio

John Blake is a research lecturer at the Japan Advanced Institute of Science andTechnology. He has taught English at universities and schools for over 20 years inJapan, Thailand, Hong Kong and the UK. His current research interest is corpus
analysis of scientific research articles. johnb@jaist.ac.jp
Received: November 18, 2014
Accepted: September 29, 2015110