Using Child Utterances to Evaluate Syntax Acquisition Algorithms

Using Child Utterances to Evaluate Syntax Acquisition Algorithms Franklin Chang (chang.franklin@gmail.com) NTT Communication Sciences Laboratories 2-4, Hikaridai, Seika-cho, Kyoto 619-0237, Japan Elena Lieven (lieven@eva.mpg.de) and Michael Tomasello (tomas@eva.mpg.de) Department of Developmental and Comparative Psychology, Max Planck Institute for Evolutionary Anthropology Deutscher Platz 6, 04103 Leipzig, Germany Abstract set of words from the utterance that we are trying to predict. This approach fits better with human performance and allows us to use corpora from multiple languages to test ideas from theories of syntax acquisition and adult sentence production. Our version of this evaluation measure will be called Word Order Prediction Accuracy (WOPA) and it is based on models of sentence production. In production, one has a message that one wants to convey. To model the effect of the message in a rough way, we start with an unordered bag of words made up of words from the utterance that we want to predict, which we call the candidate set. Given this candidate set, the system has to try to predict the order of words in the sentence in a word-by-word fashion. After a word has been produced, it is removed from the candidate set, and the system tries to predict the next word in the sequence. After a sequence has been produced in this manner, we compare the sequence against the actual utterance in the corpus, and record whether it was correctly predicted or not. The Word Order Prediction Accuracy (WOPA) score is the number of correctly predicted utterances out of all the utterances of two words or more (one word utterances only have one ordering). In this article, we will use WOPA to compare six syntax acquisition algorithms based on ideas from work in computational linguistics and child language. For input and testing, we will use utterances from child and parent interactions from twelve typologically-diverse languages. Our goal is to be able to determine which of these algorithms best matches the knowledge that may have yielded these corpora. More generally, we hope that this demonstration will suggest that WOPA can be useful as a general way of comparing computational approaches that use different internal representations. Several algorithms for learning syntactic categories from distributional information were tested against utterances from adults and children in twelve typologically different languages. The evaluation measure that was developed allows one to examine word order constraints over a whole corpus and developmentally. By comparing several different algorithms of varying abstraction against actual corpora of children’s speech, the evaluation measure determined that lexically specific knowledge is more advantageous than more broad-based category knowledge in predicting word order. Introduction There is a growing interest in unsupervised computational approaches to syntax acquisition (e.g., Mintz, 2003; Redington, Chater, & Finch, 1998). These systems collect statistical information from corpora and use that information to extract syntactic categories and constraints. These systems are evaluated by comparing their internal representations with syntactic representations that have been labeled or created by humans (e.g. tagged corpora). To compare these systems to human children, we would need to label child utterances with a set of syntactic representations. Since there is little agreement about the nature of syntactic representations in human children at each point in development (contrast Pinker, 1984, with Tomasello, 2003), it is difficult to use child utterances with computational approaches that are evaluated against human labeled representations. One way to evaluate computational models that does not depend on tagged categories is suggested by connectionist approaches to syntax (Elman, 1990). Connectionist syntax acquisition models learn internal abstractions by making predictions over words. In these systems, the accuracy at predicting the order of words in a sentence is a measure of the system’s language knowledge. One difficulty with scaling these systems up to real corpora is the use of neural assumptions such as gradual weight-based learning. In language models (e.g., n-grams), similar evaluation measures have been used, but since there are not enough constraints to predict the next word from the whole lexicon, researchers have typically looked at performance of these models with a subset of the data (e.g., frequent words). Here, we used prediction as the evaluation measure, but restrict it to the Corpora Computational learners should be tested against multiple typologically-different languages to avoid biases towards particular languages (e.g., English) or particular language typologies (e.g., languages without rich morphology). This is not usually done, because tagged corpora are not always available for different languages. WOPA does not evaluate against human-labeled syntactic tags and can be evaluated against raw word-separated corpora. Our corpora were twelve typologically diverse corpora