Bridging computational, formal and psycholinguistic approaches to language

Bridging computational, formal and psycholinguistic approaches to language Zach Solan, David Horn, Eytan Ruppin Faculty of Exact Sciences Tel Aviv University Tel Aviv, Israel 69978 {zsolan,horn,ruppin}@post.tau. ac.il Shimon Edelman Department of Psychology Cornell University Ithaca, NY 14853, USA se37@cornell.edu quences (syntagms) is the basis for the classical distributional theory of language [10], as well as for some modern works [11]. Likewise, the pattern — the syntagm and the equiva- lence class of complementary-distribution symbols that may appear in its open slot — is the main representational build- ing block of our system, ADIOS (for Automatic DIstillation Of Structure). Our goal in the present paper is to help bridge statistical and formal approaches to language [12] by placing our work on the unsupervised learning of structure in the context of current research in grammar acquisition in computational lin- guistics, and at the same time to link it to certain formal theo- ries of grammar. Consequently, the following sections outline the main computational principles behind the ADIOS model, and compare these to select approaches from computational and formal linguistics. The algorithmic details of our ap- proach and accounts of its learning from CHILDES corpora and performance in various tests appear elsewhere [1, 2, 3]. In this paper, we chose to exert a tight control over the tar- get language by using a context-free grammar (Figure 1) to generate the learning and testing corpora. Abstract We compare our model of unsupervised learning of linguistic structures, ADIOS [1, 2, 3], to some recent work in computa- tional linguistics and in grammar theory. Our approach resem- bles the Construction Grammar in its general philosophy (e.g., in its reliance on structural generalizations rather than on syn- tax projected by the lexicon, as in the current generative the- ories), and the Tree Adjoining Grammar in its computational characteristics (e.g., in its apparent affinity with Mildly Con- text Sensitive Languages). The representations learned by our algorithm are truly emergent from the (unannotated) corpus data, whereas those found in published works on cognitive and construction grammars and on TAGs are hand-tailored. Thus, our results complement and extend both the computational and the more linguistically oriented research into language acqui- sition. We conclude by suggesting how empirical and formal study of language can be best integrated. The empirical problem of language acquisition The acquisition of language by children — a largely unsuper- vised, amazingly fast and almost invariably successful learn- ing stint — has long been the envy of natural language en- gineers [4, 5, 6] and a daunting enigma for cognitive scien- tists [7, 8]. Computational models of language acquisition or “grammar induction” are usually divided into two categories, depending on whether they subscribe to the classical gener- ative theory of syntax, or invoke “general-purpose” statisti- cal learning mechanisms. We believe that polarization be- tween classical and statistical approaches to syntax hampers the integration of the stronger aspects of each method into a common powerful framework. On the one hand, the statisti- cal approach is geared to take advantage of the considerable progress made to date in the areas of distributed represen- tation, probabilistic learning, and “connectionist” modeling, yet generic connectionist architectures are ill-suited to the ab- straction and processing of symbolic information. On the other hand, classical rule-based systems excel in just those tasks, yet are brittle and difficult to train. We are developing an approach to the acquisition of distri- butional information from raw input (e.g., transcribed speech corpora) that also supports the distillation of structural reg- ularities comparable to those captured by Context Sensitive Grammars out of the accrued statistical knowledge. In think- ing about such regularities, we adopt Langacker’s notion of grammar as “simply an inventory of linguistic units” ([9], p.63). To detect potentially useful units, we identify and pro- cess partially redundant sentences that share the same word sequences. We note that the detection of paradigmatic vari- ation within a slot in a set of otherwise identical aligned se- Figure 1: the context free grammar used to generate the cor- pora for the acquisition tests described here. The principles behind the ADIOS algorithm The representational power of ADIOS and its capacity for un- supervised learning rest on three principles: (1) probabilistic inference of pattern significance, (2) context-sensitive gener- alization, and (3) recursive construction of complex patterns. Each of these is described briefly below. Probabilistic inference of pattern significance. ADIOS rep- resents a corpus of sentences as an initially highly redundant directed graph, in which the vertices are the lexicon entries and the paths correspond, prior to running the algorithm, to corpus sentences. The graph can be informally visualized as a tangle of strands that are partially segregated into bundles.