A Wikipedia-based English-Language Reference andTeaching Tool
Jason Ginsburg
University of Aizu
Tsuruga, Ikki-machi
Aizuwakamatsu, Fukushima, Japanjginsbur@gmail.com
ABSTRACTThis paper describes software under development that
automatically extracts information from corpora that can be usefulfor language learners and language researchers. This software
provides users with lists of words (extracted from several large
corpora) that are frequently used in various English genres. In
addition, the software extracts the following types of informationfrom Wikipedia: 1) word frequency: users can view lists offrequent words, 2) word forms: users can view different
morphological forms of target words, 3) part of speech
information: users can view the part of speech distribution oftarget words, and 3) example paragraphs: users can view sample
paragraphs containing different usages of target words.

and it could be used to help people in the developing world learn alanguage (English, etc.).
The organization of this short paper is as follows. In section 2, I
describe the primary materials that this software is built with. In
section 3, I explain the various functions of the software. Section
4 is the conclusion.

2. MATERIALSThis project utilizes the following corpora:
Table 1: CorporaCorpus

Size (tokens)

Keywords

English Wikipedia [3]1,003,961,037 (after cleaning)

corpora, Wikipedia, language teaching, language research

English Gigaword [2]2,698,666,444

1. INTRODUCTIONPERC Corpus [4]27,211,335

Large natural language corpora can be excellent resources forlanguage teaching, since they contain real language that alanguage learner could benefit from being able to understand. Thispaper discusses a reference tool that attempts to harness the powerof large corpora (Wikipedia in particular) as a resource for
language learners, as well as for language researchers. Wikipedia
is particularly useful because it contains language written by
many different people and it contains texts that cover a wide
variety of subjects. The reference tool described in this paper
extracts target information from corpora, stores this information,
and allows users to easily access this information.

PERC
Corpus:Computer
ScienceandEngineering
sections

3,915,319

This project has several primary goals: a) get a computer toautomatically create useful reference materials for language
teaching, learning, and research, b) create reference materials ofhigher quality than can be created without the aid of large corpora,c) create software that is easy to use and that provides useful
information to a user, and d) create a product that is language andgenre independent and thus can be utilized for various languages
and genres. With regard to d), if you have a suitable corporus (e.g.,Wikipedia) and appropriate Natural Language Processing tools fora particular language, then you should be able to automatically
reproduce this reference tool for that language. Thus, although
this reference tool is designed for English, a similar tool could be
produced for another language if there are large corpora (such as
Wikipedia) and natural language processing tools for thatlanguage.
In addition, this software could have applications for thedeveloping world. It could be made available for free on the Web,I downloaded Wikipedia and filtered out most Wikipedia tags.
English Gigaword is an enormous corpus consisting of texts frominternational English news services. The Professional English
Research Consortium (PERC) Corpus consists of texts fromEnglish academic journals in various fields of science andtechnology. PERC is comprised of 22 different corpora in various
science and technology fields. I utilized two versions of thiscorpus; 1) the complete corpus, and 2) a portion of the PERC
corpus consisting only of the Computer Science and Engineering
sections.
The main components of this software were created using thePython programming language. In addition, I utilized thefollowing software from the Natural Language Toolkit [1] toextract relevant information from Wikipedia: a part-of-speechtagger, a stemmer (the Porter Stemmer) and a WordNet
Lemmatizer.

3. FUNCTIONS
3.1 Display
A display (Figure 1), allows a user to make selections and conduct
searches for various bits of information.

determine how many words are displayed in a list. A user can thenselect a word from a list for a target genre (see Figure 2). In
addition, the “Search” box enables a user to input a target word of
his/her choice (top of Figure 2).

3.3 Search Functions
Once a user selects or inputs a word, a window appears thatprovides a user with target information about a word. Figure 3
shows a window for the word analysis.
Figure 1: Display

3.2 Word list functions
The “Genre” function (Figure 1) displays word lists, of various
genres, which were created from the corpora given in Table 1.
When a user selects a genre, a portion of the word list is shown.
The word lists for each genre were created from the relevant
corpora as shown in Table 2.
Table 2: Genres
Genre

SourceSize (types)

Figure 3: analysis

General: WikipediaWikipedia

72,081,643

General: News

English Gigaword

55,522,640

The user then can choose to view information, which is
automatically extracted from Wikipedia, about the target word.

Technical: SciencePERC

860,653

Technical:
Computer Science/Engineering

PERC
ComputerScienceandEngineering portions

101,657

Clicking on the “Frequency” button (Figure 4) displays thefrequency (and other relevant information) of the selected word,
as calculated from the Wikipedia corpus. For example, the word
reverse occurs 272 times in Wikipedia. Frequency information
was obtained by calculating the unigram frequencies for all tokens
in Wikipedia.

These word lists for various genres can be useful for students andresearchers who want to see what words tend to be used inparticular genres of English. For example, the “Technical:Science” list can be useful for English language learners who are
studying an area of science.
This software allows a user to manipulate a target word list using
various functions (Figure 1). The “Options” function allows a user
to choose whether or not to see the frequency of each word, as it
is found in the particular corpus. For example, a user can view alist of frequent words in the “General: News” genre that also
displays the frequency of each word (the frequency corresponds tothe frequency of that word in the English Gigaword corpus). The“Filter” option allows a user to choose whether or not to include
stop words (frequently occurring words such as the, a, am, are,etc.) in the word lists. The “SORT” function enables a user tochoose whether words should be presented in order of frequency
or in alphabetical order. The “SHOW” function allows a user toFigure 2: Choosing a target wordFigure 4: Frequency of reverse
The “Word Forms” button displays various forms of a word
(Figure 5). This information is automatically extracted fromWikipedia in the following way. All words in Wikipedia werestemmed with the Porter stemmer and lemmatized with theWordNet lemmatizer. Then all words that have an identicalstem/lemma were grouped together to create lists ofmorphological forms of each word in Wikipedia. Note that forwalk, the algorithm finds some erroneous forms, such as walke,
but this results from the fact that walke appears in Wikipedia(since Wikipedia is so large, erroneous word forms, resulting fromFigure 5: Word Forms for walk

misspelling, etc. appear).
The “Part of Speech” button displays the Part of Speech (POS)
distribution of target words. This function is designed to show thefrequency of the various POS categories for a target word, wherePOS labels and frequency of each POS label are automaticallyextracted from Wikipedia. In order to calculate this information,
the entire Wikipedia corpus was tagged with the Natural
Language Toolkit built-in POS tagger. Then I created a file that
contains the POS frequencies for each tag for a particular token.
For example, the POS distribution of the word walk is shown inFigure 6.
Figure 8: Examples: cause as a Verb
Wikipedia corpus, there are many example paragraphs for any
frequent word.

4. CONCLUSION

Figure 6: Part of Speech distribution for walk
The “Examples” button presents a user with paragraph length
texts containing target words. This function is unique.
Concordancers are used for corpus research and language teaching.
They provide at most sentence length examples. However,Wikipedia enables extraction of larger paragraph length examples.
Examination of a paragraph enables a user to get a better idea of
how a target word is used, since there is more natural language
discourse presented to a user than would be presented with asentence length example.
Example paragraphs are extracted in the following way. I createda file tha emulates a dictionary data structure, with keys andvalues. The key consists of a token and its part-of-speech tag; e.g.,(‘token’, POS). Each key has a value, which is a list of allpositions in all texts in which the particular token occurs with thatparticular POS tag; e.g., Value = [‘1.txt’,’Line_29’],[‘17.txt’,’Line_518’, Line_519’], etc. An advantage of Wikipedia
is that it has line breaks between paragraphs. This enables easy
extraction of sample paragraphs. For example, Figure 7, shows anexample paragraph for the word cause (which appears in bluetext) with the POS tag Noun (N). Figure 8 shows an example
paragraph for the same word with the POS tag Verb (V). A user
can view further examples for a target word by clicking on thedesired POS tag. Since examples are extracted from the enormous

This software provides frequency-based word lists, frequency
information about target words, morphological forms of targetwords, POS information about target words, and examples
containing target words. The current functions require further
refinement and additional functions need to be added, such as acollocation finder and dictionary. In addition, the accuracy of theinformation presented in this software needs to be evaluated. I
eventually hope to make this system available for students
studying English.
Lastly, this research has applications for a variety of fields. It hasapplications for applied linguistics, since the purpose is to develop
materials that can be used for teaching and learning languages. It
has applications for theoretical linguistics, since it can be used forinvestigating parts of speech, morphology, semantics, discourse
analysis, etc. It has applications for computer science (e.g., naturallanguage processing) since it requires development of algorithms
to extract natural language information from corpora. It also hasapplications for artificial intelligence, since a goal is to get acomputer to automatically create useful reference and researchresources - automatic creation of reference materials could insome sense be likened to ‘intelligence’. Further research anddevelopment remains to show how much this project cancontribute to these fields.

5. ACKNOWLEDGMENTS
Thanks to the audience of the ACM Chapter Conference on
eLearning and Technical Documentation for the Developing
World for helpful comments and questions. I also thank the PERC
Corpus project team for allowing me to use the PERC corpus.

6. REFERENCES[1] Bird, S., Klein, E., and Loper. E. 2009. Natural Language
Processing with Python. O’Reilly, Sebastopol, CA.
http://www.nltk.org/[2] Graff, D., Kong, J., Chen, Ke, and Maeda, K. 2007. English
Gigaword Third Edition. Linguistics Data Consortium,
Philadelphia.
[3] http://en.wikipedia.org/wiki/Wikipedia:Database_download#
English-language_Wikipedia
Figure 7: Examples: cause as a Noun[4] http://www.corpora.jp/~perc