*Ontolex***:** An Extensible, Internet-based Framework for Modeling
Second Language Acquisition

Fortuna, Emily^1,\ 2^ and Reeker, Larry ^1^

^1^ National Institute of Standards and Technology

^2^ Rice University

**Abstract**

> *Since the birth of Computer Science and modern Linguistics, numerous
> cognitive computer models have emerged with a myriad of approaches as
> a method to study the mysterious black box of the human mind. This
> project is the second generation of an effort to create an extensible
> reference model for these models, or a meta-model of second language
> learning. The internet-based framework, called Ontolex, is an effort
> to abstract ontology design and the ideas of several grammar theories
> in order to universalize its application and extensibility and
> eliminate dependency on any specific natural language. Ontolex creates
> word-to-ontology relationships via a three tier system mapping words
> to lexemes to concepts in the ontology. This learning system, part of
> an ongoing implementation, will model three general phases in the
> human second language learning process: vocabulary building, formulaic
> speech and translation, and unique utterance generation with user
> reinforcement feedback. *

**1. Introduction**

**1.1 Background**

Since publication approximately fifty years ago, Noam Chomsky\'s
linguistic ideas have sparked a watershed of fresh theories in
linguistics. His new ideas focused first on syntax and then increased
scope to encompass semantics, previously considered an area for
lexicographers and philosophers. This sparked many different theories of
how languages use syntax with semantics to produce informative
utterances and how these utterances are understood by other people.

At the same time as Chomsky's early publications, computer scientists
(notably Simon and Newell \[16\]) were developing the early ideas for
Computational Science, the use of computers to provide computational
models to test theories. Many of the computational models were attempts
to model possible cognitive processes, giving rise to the branch of
science called cognitive science, which studies cognitive theories via
cognitive modeling.

Chomsky's transformational grammar theory used "immediate constituents
analysis," developed by Rulon Wells \[24\]. The technique consists of
creating a tree structure in which the utterance is divided into phrases
that would be syntactically correct if each phrase were replaced by a
single word, then similarly recursively decomposing each phrase until
reaching individual words at the end of the tree \[18, 24\]. Chomsky's
transformational grammar approach to analyzing language applied easily
to computer linguistic analysis because of its similarity to techniques
needed to parse syntax of high-level computer languages into machine
language. Sentences using the same words to convey the same meaning, but
in different orders, such as "The dog ate the bone" and "The bone was
eaten by the dog" are parts of the same "deep structure," a structure to
represent the underlying meaning, according to Chomsky. This approach
provides a very systematic, regular method of parsing sentences,
however, more complex sentences, sentences composed in languages other
than English, and imperfectly formed utterances do not easily fit into
this model. In particular, null subject languages, in which the subject
can be omitted, like Spanish and Japanese cause particular difficulty
specifying an initial noun phrase \[25\].

Lexical functional grammar, head driven phrase structure grammar, and
construction grammars developed an alternate method for analyzing the
parts of a sentence in which utterances are mapped in an attribute-value
matrix (AVM) \[9, 20\]. The Lexical Functional Grammar (LFG) approach
constructs the AVM, or feature structure, into a matrix containing five
attribute names: SUBJ, TENSE, NEG, PRED, and COMP \[4\]. SUBJ and COMP
are "functions," consisting of smaller AVMs. LFG sees free constituent
order languages such as Japanese, Malayalam, Warlpiri, and Wambaya as an
argument against the transformational grammar proposed by Chomsky \[4\].
Construction Grammar theories derive from the idea that the underlying
meaning of an utterance is rarely a simple sum of the meaning of its
individual parts. This theory attempts to encompass all utterances,
perfectly formed or otherwise \[9\].

Around the same time a child learns words in his or her first language,
the child begins to form an ontology \[8\]. An ontology formally defines
the relationship between concepts in a domain \[17\]. Ontologies have
many forms, and one might visualize them as a web or hierarchy of
references \[22\]. During second language learning, an individual's
ontology has already been formed and the learner simply associates the
new words with the preexisting ontology. Second language learning is an
ideal candidate for study of language acquisition because the ontology
formation and language acquisition processes are separated.

Research to integrate ontologies with natural languages is promising.
The Semantic Web project envisions setting up "common formats for
interchange of data" between computers and people \[12, 15\]. Advances
in this project have relevance to linguistics \[10\] and to our interest
in language learning modeling. Since the nineties, Charles Fillmore has
continued to develop FrameNet, an online database that stores and
visually presents the combinatory possibilities of English words through
"frames" \[6, 7, 15\]. An extension of the FrameNet project for the
Spanish language is currently under development \[23\]. Other
researchers recently developed a statistical grammar model for German by
developing a context free grammar and using statistical analysis to
ascertain the frequency of constructions \[21\].

**1.2 Research Goals**

The purpose of this project is to create an easily extensible,
web-accessible, and natural language-independent cognitive model of
second language learning that maps words in the native and foreign
languages to a single, underlying ontology. Through expansion, this
framework can serve as a testing ground for future theories in second
language acquisition and as a comparison with other models. It is the
second generation of an intended series of "Basic Reference Models"
\[26\], and invites other researchers to build upon its framework.

In addition, created with the goal of language abstraction in mind
\[1\], the model uniquely combines the principle of Occam's razor with
ideas inspired by lexical-functional grammar and construction grammars.
The model, called *Ontolex*, focuses on second language learning through
the written word and assumes that the individual has already developed a
simple hierarchal ontology of concepts. *Ontolex* does assume that a
linguistic ontology is independent of the natural language. This initial
model combines several proposed phases in second language learning
\[11\]. The intelligent model combines phases of second language
learning into three general groups: vocabulary building, formulaic
speech and translation, and unique utterance generation with user
feedback.

**2. Architecture**

*Ontolex* consists of three general parts modeling the language learning
process: vocabulary and ontology building, translation and form
sentences, and unique sentence generation. The entire system was
developed with language and syntax independence in mind and can easily
be extended for any new natural language.

**2.1 Vocabulary and Ontology Building**

This section allows the user to enter, edit, and delete entries in
*Ontolex*'s linguistic knowledge database. The first subsection, the
Ontology Builder and Viewer, provides an illustration of the system's
current ontology in a tree form, and allows the user to delete and enter
in new concepts at any location in the hierarchy.

The Sentence Learner section allows the user to build up the syntactic,
lexical, and semantic vocabulary of *Ontolex* by entering in complete,
declarative sentences. With only slight modification of the GUI and
Control layers (not yet implemented), the user can enter in
interrogative and exclamatory sentences, as well as phrases that
constitute a complete utterance.

To use the Sentence Learner, the user selects the foreign language and
the native language and types in the foreign and native sentences. In
this initial vocabulary building phase of language learning, the user
gives *Ontolex* basic information regarding the part of speech of
elements in the sentence and the concepts to which the parts of the
sentence correspond. In this manner *Ontolex* has the capability to map
idioms to concepts in addition to individual words. The sentence
structure and substructures---the elements that make up an acceptable
Subject construction for example---are saved in the database. The words
of the foreign sentence are saved into the database linked to the
concept. Figure 1 shows a screenshot after the user has entered the
sentence and must enter in the more specific data regarding the terminal
tokens in the sentence. The Database Editor and Viewer allow the user to
view the words, lexemes, and sentence structures that have been saved in
the database for a particular language.

**2.2 Translation and Form Sentences**

The intent of the Translation and Form Sentences section is to model the
stage in language learning in which the learner understands sentences by
translating them into words in his or her native language. Although a
graphical user interface has been developed for this section, the logic
for this section of *Ontolex* is currently theoretical.

Under the current plan, the user will type in a sentence, identifying
the parts of speech. *Ontolex* will then attempt to maneuver from
foreign words to concepts and then back up to words in the native
language. Once the lexeme database becomes large enough, Ontolex will
begin to be able to identify parts of speech by matching the words in
its database. If *Ontolex* encounters a foreign word that it has not
encountered before, it will ask the user of its ontological meaning
(much like a student learning a new word), and then link words to
concepts accordingly.

**2.3 Unique Sentence Generation**

![](media/image1.png){width="3.7465277777777777in"
height="3.473611111111111in"}The Unique Sentence Generation section
represents the phase when the learner has reached a more advanced stage
in learning the foreign language. In this phase the learner can relate
foreign words directly to concepts and vice versa. Ontolex is currently
designed for two methods of sentence generation: by specifying semantics
and specifying syntax.

In the Semantic Specification method (Figure 2 next page), the user
specifies the concept for the verb and up to three noun ideas to be
incorporated into the syntactically and semantically correct simple
sentence created dynamically. For the Syntactic Specification method,
which is currently theoretical, the user types in the desired syntax for
a sentence and selects a language. *Ontolex* generates what it believes
to be a semantically correct sentence using a recursive method similar
to the algorithm used for Semantic Specification. *Ontolex* saves the
entered sentence structure and substructures into its database for
future use. On select random sentences, *Ontolex* will attempt to
generalize concepts by selecting a parent class rather than the typical
corresponding class for a verb subject pair. Based on user feedback,
Ontolex can determine if it effectively generalized or produced an
overgeneralization error \[12\], typical of new language students.

In this manner, *Ontolex* can utilize the full capability of the
ontology hierarchy. For example, if *Ontolex* knows that "I eat an
apple" is a valid combination of elements, then it may attempt to
generalize to see if "I eat an orange" is also a valid combination. If
the user affirms the validity of this combination, Ontolex can
generalize that perhaps all fruits can be placed in the object slot with
the combination "I eat \_\_."

**3. Implementation**

Although the graphical user interface has been developed for all three
parts, currently the underlying control logic has been implemented only
for the vocabulary and ontology building and the Semantic Specification
of the unique sentence generation portions. The program was created in
Python due to its conciseness, ease of maintenance, and ease of
extensibility for future work. It also uses the SQLObject Python module
to interface to the SQLite database. The code also produces valid XHTML
1.0 for the webpage. In its current stage of implementation, *Ontolex*'s
focus has been on implementing *capability* for expansion, rather than
filling a database with a well developed ontology and lexicon. As a
result, *Ontolex*'s ontology, lexeme, and word bank are tentative
entries primarily for testing *Ontolex*'s abilities. *Ontolex* can be
found online at http://ontolex.nist.gov.

![](media/image2.png){width="3.375in" height="2.8555555555555556in"}

**3.1 Ontology Implementation and Its Relation to Words**

As envisioned for this project, an individual's ontology is a web-like
hierarchy of concepts -- underlying meanings behind words, phrases, and
sentences -- that exist in his or her brain, independent of natural
language. Ideally, items in the ontology would be represented by mental
images or the ideas behind thought processes. However, since researchers
have yet to develop computers with human brain waves, a concept in
*Ontolex*'s ontology is represented as an English word surrounded by
square brackets \[\].

Although the researchers of this project realize that "real" ontologies
can be potentially cyclical, the ontology structure represented in this
framework is a simple hierarchy of concepts, as shown in Figure 3.
Concepts can have zero or more children.

The ontology structure itself in *Ontolex* deviates from the typical
solely noun-based ontology \[17\] or an ontology dependent on a single
part of speech \[3\], and attempts to represent all parts of speech
deriving from a single idea within a single ontological entry. For
example "to teach" and "teacher" both correspond to the ontological
concept \[teach\]. Reinforcing the unique nature of concepts, a concept
in the hierarchy cannot share a name with any other concept.

*Ontolex*'s database are linked to their corresponding concepts by means
of a three-layer system (shown in Figure 3): concepts, lexemes, and the
words in the lexicon. A lexeme is a base linguistic unit in a language
from which a variety of allomorphs can be derived. For example, Figure 4
on the next page shows the different forms "imagining," "imagined,"
"imagines," and "imagine" all derive from the same base lexeme. The
diagram shows a close up of a single node in the ontology tree and its
relation to the lexeme and word layers. A variety of synonyms in the
lexeme layer all correspond to a single concept in the ontology.
Elements in the lexeme layer from different languages may correspond to
the same concept. Lexemes are represented by words in the specific
language surrounded by parentheses. For each item in the lexeme layer, a
variety of forms of the single word can be derived for the word layer.
Although some elements are saved in the database as members of the
lexicon (word) layer, others are created from items in the lexeme layer
by language-specific conjugation and inflection functions. Through these
functions *Ontolex* develops a general sense of the grammatical rules of
a particular language, which it may use later for sentence generation.
By storing basic rules of a language in the database, the system will
follow the same patterns of overgeneralization in which individuals
learning a second language engage \[13\]. In subsequent versions of this
project, the word layer will not have any entries in the database, and
the conjugation and inflection functions will simply create words on the
fly.

An element in the lexeme layer holds information about its part of
speech, the concept(s) to which it corresponds, what parts of speech
modify (add meaning to) it, ways its use deviates from the general
grammar rules of their language ('grammar exceptions'), and any other
attributes pertinent to a particular part of speech (such as possible
valencies for a verb). For example, using the conjugation and inflection
functions *Ontolex* may produce the phrase "I thinked" until the user
corrects the system through reinforcement feedback. This feedback
regarding irregularities of a particular word in a language is stored in
the 'grammar exceptions' slot (not yet implemented). In this manner
*Ontolex* will progressively learn additional, word-specific rules
regarding the language. The researchers of this project also envision
developing a process that collects the word-specific grammar exceptions
and attempts to formulate additional *meta-*rules through inductive
reasoning.

A word in the lexicon layer holds information regarding its part of
speech, the lexeme(s) to which it corresponds, and other pertinent
attributes specific to its part of speech in that language (for example:
person, number, tense, aspect, and mood for a verb). To circumvent
issues with homonyms, the attribute storing the corresponding lexeme is
the lexeme's *entry number* in the database, rather than its name.

**3.2 Linguistic Theory Design Implementation and *Ontolex* Algorithm
Specifics**

Initially Chomsky's transformational grammar method was considered for
design implementation but was ultimately rejected because it did not
allow for sufficient abstraction and language independence. Maneuvering
between Chomskian tree structures of two different languages required
too much one-to-one rule writing for each case. The grammar implemented
here incorporates ideas from lexical-functional grammar, and
construction grammars, without relying on characteristics of a specific
language. Instead, we pursued a more general grammar design.

For entering declarative sentences into *Ontolex*'s Sentence Learner,
the user divides the sentence into up to four of the overarching
constructions using parentheses (see box to right). These terms are used
very loosely: "O" does not always correspond to a direct object in the
truest sense of the word. Instead, "O" refers to a part of the sentence
that completes the meaning of the verb, such as "red" in "The rose is
red." Within that enclosed phrase the user continues to surround words
with parentheses according to the more specific elements or
constructions as necessary until a terminal unit is reached (this is
usually just a word, but may be several for idioms). A terminal unit,
determined by user discretion, is a word or a set of words in which the
meaning of its parts is drastically different from the meaning of the
combination of its units. For example the Spanish phrase *echar de
menos* is considered idiomatic because literally it translates to the
nonsensical English phrase *to throw of less* but really means *to miss*
\[something\]. Entering data in this manner allows the user to enter
idioms and not just word-for-word translations.

A verb's valency is the number of arguments it takes to form a complete
statement. The verb is classified as intransitive, transitive,
ditransitive, or having zero valency based on its number of arguments.
Only a null subject language like Spanish can have a verb with a valency
of zero, although a "doing" agent ('S') is still present, through the
conjugation of verb. An example of a Spanish verb exhibiting zero
valency is the impersonal, undefined subject "it" when using the verb
*nevar*, "to snow," often conjugated *nieve*, "it is snowing."

At the heart of the current implementation of Sentence Generation via
semantics, a simple sentence consists of a verb phrase ('VP') and the
agent corresponding to the verb (abbreviated 'S'). Based on the number
of units of meaning the user provides, *Ontolex* searches in its syntax
database for a sentence with a corresponding number of parts. First it
selects a sentence subject from the selected noun concepts based on
compatibility with the verb lexeme. If the noun's concept is an equal or
child class of one of the allowed subject modifiers specified by the
verb lexeme, that noun can act as a subject of the sentence. If no
specified noun seems to be compatible with the selected verb lexeme,
then the system randomly selects a noun from the group, and relies on
user reinforcement feedback (not yet implemented) to subsequently save
this new concept as compatible with the verb. In this manner *Ontolex*
"discovers" new, potentially valid combinations of words that it has
never before encountered, similar to the way humans constantly create
new sentences never before uttered. *Ontolex* continues recursively
parsing through the elements of the sentence to select matching articles
for nouns, etc. based on inflection and conjugation functions (not yet
implemented) specific to the language. With one set of inputs, *Ontolex*
can generate several syntactically and semantically correct sentences.

**3.2 Overall Application Framework**

Ontolex was constructed with the Model-View-Controller design pattern in
mind, in order to maximize extensibility for the framework. Figure 5 on
the next page highlights the interaction between the actual elements in
the framework.

**4. Conclusions and Future Work**

We presented an extensible, relatively language independent web-page
framework for testing theories in second language acquisition that
relate an ontology to a lexicon. Our implementation has shown that it is
possible to model several phases in the language learning process by
mapping words to an ontology. The Sentence Generator by specifying
semantics has produced encouraging results. We plan to implement the
remaining components and expand the ontology and lexeme database.

Given the favorable performance of our current software framework, we
believe that it is possible to create a simple universal model for
grammar that is not language dependent. As our grammar
![](media/image6.png){width="6.451574803149606in"
height="4.3728346456692915in"}*Ontolex* will be prepared to handle
extreme variations in grammar. Universal grammatical models for
converting from one language to another are definitely a topic worthy of
future investigation and possible implementation in *Ontolex*.

An additional issue that has come to light in this study and has
hindered the development of *Ontolex* is the need for standards in
developing ontologies. Unless standards for a universal linguistic
ontology are developed Computer Scientists will forever be creating
ontologies for ontologies, like interfaces for interfaces, so that
different systems can communicate with one another. Subsequent versions
of this project intend to have a more complex ontological structure,
perhaps in which concepts have attributes detailing specific
characteristics of the concept.

It is well known that while reinforcement learning may play some role in
language learning, it does not significantly contribute to learning a
grammar during language acquisition. Therefore, alternate methods of
learning aside from reinforcement learning in the Sentence Generation
phase, such as case-based learning, also merit exploration. We would
like to add the capability of subject and part of speech identification
in the first and second phases of the framework to more accurately model
language learning \[2\]. In addition, statistical analysis in
determining language structures holds potential for investigation.

Implementing a more user-friendly GUI will improve the accessibility of
the application. We would like to create a more readable ontology view
for large ontologies and an easier method for entering in sentences in
the Sentence Learner phase. Lastly, we believe investigating a
metalanguage for entering rules of a natural language and letting the
computer develop conjugation and inflection functions, rather than
manually coding the conjugation and inflection rules, holds potential
for exciting future development.

**Acknowledgements**

Thanks to Dan Sandler, Michael Benza, Christopher Warrington, and the
Python Tutor mailing list for programming language implementation
discussion and suggestions in the early stages of this project. We also
thank the Information Technology Laboratory at the National Institute of
Standards and Technology for funding this research.

**References**

\[1\] "Babel's Children," *The Economist Newspaper and The Economist
Group*.
http://www.economist.com/science/displayStory.cfm?story_id=2329718, Jan.
8, 2004.

\[2\] Chomsky, N., *Noam Chomsky on the generative enterprise: A
discussion with Riny Huybregts and Henk van Rimsdijk*, Dordrecht: Foris,
1982.

\[3\] Everett, J. O., Bobrow, D. G., Stolle, R., Crouch, R., de Paiva,
V., Condoravdi, C., van den Berg, M., Polanyi, L., "Making Ontologies
Work for Resolving Redundancies Across Documents," *Communications of
the ACM*, 45 \[2\], pp. 55-60, 2002.

\[4\] Falk, Y. N., *Lexical-Functional Grammar: An Introduction to
Parallel Constraint-Based Syntax*., Stanford: Center for the Study of
Language and Information, 2001.

\[5\] Farrar, S., Lewis, W. D., & Langendoen, T. D., "A Common Ontology
for Linguistic Concepts," *Proc. of the Knowledge Technology
Conference,* Seattle, Mar. 2002.

\[6\] Fillmore, C. J. *FrameNet*. Berkeley, CA: University of
California, Berkeley, Department of Linguistics.
http://framenet.icsi.berkeley.edu/, July 2006.

\[7\] Fillmore, C. J., "Verbs of Judging: an exercise in semantic
description," In Fillmore & Langendoen (eds.) *Studies in Linguistic
Semantics*, New York: Holt, Rinehart & Winston, pp. 272-289, 1971.

\[8\] Fischer, C., & Gleitman, L. R. "Language acquisition," In H.F.
Pashler (Series Ed.) and C. R. Gallistel (Volume Ed.), *Stevens'
Handbook of Experimental Psychology, Vol 1: Learning and motivation*,
New York: Wiley, pp. 445-496, 2002.

\[9\] Fried, M., & Östman, J.O., *Construction Grammar in a
Cross-Language Perspective*, Amsterdam: John Benjamins Publishing
Company, 2004.

\[10\] Gupta, K. M., Aha, D. W., Marsh, E., & Maney, T., "An
Architecture for Engineering Sublanguage WordNets," *Proc. of the First
International Conference on Global WordNet*, Mysore, India: Central
Institute of Indian Languages, pp. 207 -- 215, 2002.

\[11\] Haynes, J., "Stages of Second Language Acquisition," Everything
ESL.net, http://www.everythingesl.net/inservices/language_stages.php,
2005.

\[12\] Herman, I., "W3C Semantic Web Activity," World Wide Web
Consortium, http://www.w3.org/2001/sw/, Jul. 2006.

\[13\] Jackson, D., Constandse, R. M., & Cottrell, G. W., "Selective
attention in the acquisition of the past tense," *Computational
Cognitive Modeling: Source of the Power, AAAI-96 Workshop*. Portland,
Aug. 1996.

\[14\] Knuth, D. E., "Semantics of context-free languages,"
*Mathematical Systems Theory.* 2 (2), 127---145, 1968.

\[15\] Narayanan, S., Fillmore, C. J., Baker, C. F., Petruck, M. R. L.,
"FrameNet Meets the Semantic Web: A DAML+OIL Frame Representation," *In
Proceedings of the Eighteenth National Conference on Artificial
Intelligence,* Edmonton, CA, 2002.

\[16\] Newell, A., & Simon, H. A., "Computer Simulation of Human
Thinking and Problem Solving" In M. Greenberger (Ed.), Management and
the computer of the future New York: Wiley, pp. .94-133, 1961.

\[17\] Noy, N. F., & McGuinness, D. L., "Ontology Development 101: A
Guide to Creating Your First Ontology," Stanford: Stanford University.
http://protege.stanford.edu/publications/ontology_development/ontology101-noy-mcguinness.html,
May 2006.

\[18\] Ojeda, A. E. "Discontinuous Constituents," Davis: University of
California, Davis, Department of Linguistics.
http://linguistics.ucdavis.edu/FacultyPages/aeojeda/DiscCons.pdf, 2006.

\[19\] Pollard, C., & Sag, I. A., *Head-Driven Phrase Structure
Grammar*. Stanford, CA: Center for the Study of Language and
Information, 1994.

\[20\] Pollard, C., & Sag, I. A., *Information-Based Syntax and
Semantics*. Stanford, CA: Center for the Study of Language and
Information, 1987.

\[21\] Schulte im Walde, S., "The German Statistical Grammar Model:
Development, Training, and Linguistic Exploitation," Universität
Stuttgart, http://citeseer.ist.psu.edu/schulteimwalde00german.html, Dec.
2000.

\[22\] Sowa, J. F. *Knowledge Representation: Logical, Philosophical,
and Computational Foundations*. Pacific Grove, CA: Brooks Cole
Publishing Co. http://www.jfsowa.com/ontology/index.htm, 2000.

\[23\] Subirats, C. & Petruck, M. R. L., "Surprise: Spanish FrameNet!"
In E. Hajicova, A. Kotesovcova & Jiri Mirovsky (eds.), *Proceedings of
CIL 17*, Prague: Matfyzpress, 2003.

\[24\] Wells, R., Immediate Constituents,  *Language* 23:81-117, 1947;
Reprinted in *Readings in Linguistics*, Hamp, Joos *et al*, Latest
Version 1995.

\[25\] White, L., *Second Language Acquisition and Universal Grammar*,
Cambridge, UK: Cambridge University Press, 2003.

\[26\] Wu, E., & Reeker, L. A Computational "Basic Reference Model" for
Language Learning. *Abstract in Book of Abstracts, IPSI-Venice 2005.*
Venice, IT, Nov 10-13, 2005.