Language Learning Through Similarity-Based Generalization

The idea that language can be learned and processed based on probabilistic information has been the subject of much criticism. Principal among these criticisms is the issue of data-sparsity: it is claimed that the amount of data required to reliably estimate the parameters of any useful probabilistic model far outstrip the amount of language exposure any person could reasonably receive, not just in childhood, but in an entire lifetime. The most acute form of this objection involves unseen events: in any sample of language to which a learner is exposed, many legitimate linguistic constructions will not occur, because language is such a complex, productive system. Simple probabilistic models based on the principle of maximum likelihood will assign these events a probability of 0, incorrectly implying that they are (probabilistically speaking) impossible. In this paper we review the standard arguments regarding data-sparsity and the various methods of probability smoothing that have been proposed in the field of natural language processing in order to address them. We then propose a similarity-based model for estimating bigram probabilities in language, based on psychological principles of similarity-based generalization. In a series of 3 computational simulations we show that this method is capable of learning bigram probabilities more efficiently than other methods, and thus results in considerably improved language modeling performance. We argue that the method we propose, as well as being extremely competitive in engineering terms, also provides a powerful way of ameliorating the problems of data-sparsity, and goes some way to showing how the influential criticisms that have been levied at probabilistic models of human language learning might be solved in a manner consistent with basic psychological principles. We argue that these results, in conjunction with the ubiquity of similarity-based mechanisms in cognition, support the hypothesis that human language learners could in fact exploit similarity-based information when acquiring linguistic structure, and that there is no in principle reason to believe that language is not learned and processed through the exploitation of probabilistic information. DRAFT MANUSCRIPT – PLEASE DO NOT CITE OR CIRCULATE 2 One of the most impressive and difficult to explain abilities that language users possess is their capacity for understanding and producing a seemingly limitless number of utterances, many of which they will never have encountered before. How are people able to do this? Proposals regarding the representations that underpin human language learning in order to allow for this fall into two broad categories. In one, it is proposed that language is based upon a complex system of recursive rules which govern the basic syntax of language, and from which the multitude of surface-forms can be derived or constructed (see Chomsky, 1957, 1965, for some of the earliest and most influential statements of this idea). In the other, language is treated as being based upon probabilistic (or quantitative) propensities of varying strength, that are learned through conditioning on elements of the environment, and in particular on previously observed samples of language (see Skinner, 1957, for one early treatment of this kind of idea, and Chater and Manning, 2006, for a more recent formulation of this kind of probabilistic approach). The latter proposal, that natural language can be understood as a fundamentally probabilistic or quantitative process, and that linguistic production and comprehension can be cast in terms of learned expectancies on the part of language users has been subjected to many attacks, perhaps the most notable being mounted by Miller and Chomsky (1963; but see also Pinker and Prince, 1988; Pinker, 1994; Gold, 1967). Indeed, the impact of these objections has been so marked that a great deal of recent research into language proceeds on the assumption that language is fundamentally unlearnable, and that therefore a considerable number of the principles that govern it must somehow be innately specified (Chomsky, 1965, 1986; Pinker, 1984, 1994). In spite of the received reservations regarding probabilistic accounts of language, in recent years there has been a renewal of interest in this approach that is attributable to a variety of causes. These include the manifest difficulty of specifying complete generative grammars for natural language; an increased awareness of the lexical-specificity of many patterns in language that sit less naturally in traditional rule-based syntactic theories (Culicover, 1999; Tomasello, 2000; Goldberg, 2003); an increased appreciation of the probabilistic learning mechanisms available even to young children (Saffran, Aslin and Newport, 1996; Saffran, Newport and Aslin, 1996; Aslin, Saffran and Newport, 1998); and also the increasing application of large-scale computational methods to linguistic analysis (see Manning and Schutze, 1999, and Charniak, 1993, for reviews of such methods). Although the sophistication of probabilistic models of language has increased considerably over the last few decades, when it comes to considering models of this kind as serious models of human language acquisition, they still seem to be fundamentally plagued by the problem of data-sparsity (especially when syntactic phenomena are under consideration): the objection that in order to train a probabilistic model of language sufficiently complex to capture more than the The authors would like to thank Ewart Thomas especially for his input and support throughout the course of this research. Many thanks are also due to Gordon Bower, Nicolas Davidenko, Ulrike Hahn, David Ho, Adam November, Jon Winawer and Nathan Withoft for comments and suggestions they have made at various points. Daniel Yarlett was supported by a Stanford Graduate Fellowship. This material is based upon work supported by the National Science Foundation under Grant Nos. 0547775 and 0624345 to Michael Ramscar. DRAFT MANUSCRIPT – PLEASE DO NOT CITE OR CIRCULATE 3 most trivial of patterns, vastly more data would be required than can be collected for most practical purposes (and even if the model can be trained on a computer, the amount of data required to do so will nevertheless outstrip that available to human learners by many orders of magnitude, thus undermining the models claim to be a model of human language learning). Indeed, for many theorists, the basic problem of data-sparsity has been accepted as a proof of the impossibility of a probabilistic account of language and its learnability (Gold, 1967; Pinker, 1994). For example, the class of so-called poverty of the stimulus arguments, perhaps the most convincing and frequently advanced arguments in support of the idea that language is unlearnable from the input, are essentially arguments from data-sparsity. A poverty of the stimulus argument typically asks one to consider some syntactic phenomenon, S, and some children of age X. It is argued that the chance of a child of age X encountering an instance of S in their linguistic input is vanishingly small – essentially there is a problem of data-sparsity for children of age X with respect to S – but that nevertheless children of age X reliably exhibit sensitivity to S either in comprehension or production. The conclusion drawn is that S cannot have been learned by the children in question through induction on the input (because it was never encountered), and that therefore knowledge of S must somehow be innate in the children. But there are (at least) two grounds for objecting to this form of data-sparsity argument. (1) Is it really true that children of age X never encounter S, whatever it may be? (2) Even if S is never encountered by children of age X, is it not still possible that there are some other constructions, S′, that the child has experienced and from which children can generalize in order to show sensitivity to S? Objection (1) revolves around the status of the claims about data-sparsity and, in some of the most common forms of poverty of the stimulus arguments these claims appear, upon closer inspection, to be rather dubious (see Pullum and Scholz, 2002, for some exemplary scholarship concerning this issue). Objection (2) revolves around the mechanisms of generalization available to children in language learning, an aspect of language learning that is strangely often overlooked. Commonly, poverty of the stimulus arguments make the strong assumption that children are incapable of generalization, and that therefore they can only learn about a construction from examples of it and it alone. However, the claim that children are entirely incapable of generalization is a questionable assumption that is at odds with much research (see, for example, Ramscar and Yarlett, 2007; Maslen, Theakston, Lieven, Tomasello, 2004; Pullum and Scholz, 2002; Seidenberg and MacDonald, 1999; Rumelhart and McClelland, 1986; Berko, 1958). Moreover, even if children’s ability to generalize turns out to be relatively impoverished, it is still possible that the definition of evidence deemed relevant to S is excessively limited, and excludes much relevant information that is available in the input and can be used by even relatively naive learners. Based on this analysis, it seems to us that much headway can potentially be made against data-sparsity arguments by considering more closely the types of generalization that children may be able to engage in. In what follows, therefore, we describe the structure of traditional probabilistic language models and how they fall foul of the problem of data-sparsity. We then present a review of previous methods of ‘smoothing’ – engineering methods that have been proposed in the field of natural language processing in order to combat data-sparsity – explaining the rationale for these methods and their strengths and weaknesses. On the basis of this analysis, we DRAFT MANUSCRIPT