What grammars tell us about corpora: the case of reduced relative clauses

We present a large (65 million words of Wall Street Journal) and in-depth corpus s tudy of a particular syntactic ambiguity to investigate (1) to what extent the structure of a grammar is reflected in a corpus, and (2 )how probability flmctions defined according to a grammar fit independently established measures of syntactic disambiguation preference. We look at the well-known case of the ambiguity between a main clause and reduced relative construction. We measure the probabili ty distributions of several linguistic features (transitivity. tense, voice) over a sample of optionally intransitive verbs. In agreement with recent resuits on parsing with lexicalised probabilistic grammars (Collins, 1997; Srinivas, 1997), we find that statistics over lexical, as opposed to structural , features best correspond to human intuitive . judgments and to experimental findings. These results are enlightening to investigate novel uses of corpora, by assessing the portabil i ty of statistics across tasks, and by determining what is needed for useful syntactic annotat ion of corpora. 1 I n t r o d u c t i o n Most linguistic work until the 1950s studied language use. which required attention to detail and exceptions, and led to the development of data-driven theories and to the use of corpora to model naturally occurring language. Later on. linguists mostly studied grammars, which focussed on generalities and regularities, and led to the formulation of strong theories and to the s tudy of similarity across languages. Some of the current "empirical" approaches integrate tlle corpus-based lessons with the depth of insight that the s tudy of grammar has brought to the s tudy of language. S u z a n n e S t e v e n s o n Dept of C o m p u t e r Science and Center for Cogni t ive Science ( R u C C S ) Ru tge r s Univers i ty C o R E Building, Busch C a m p u s New Brunswick. NJ 08903 U.S.A. s u z a n n e ~ r u c c s , r u t g e r s , edu Empirically-induced models that learn a linguistically meaningflll grammar (Collins, 1997) seem to give tile best practical results in statistical natural language processing. One of the reasons wily these models perform so well compared to probabilistic context-free grammars is that they incorporate detailed lexical knowledge at all points in tile derivation (Charniak, 1997). At the same time they perform bet ter than string-based approaches because they retain structural knowledge, such as phrase structure, subcategorization and long distance dependencies. So they are equally capable of modelling the fine lexical idiosyncrasies and tile more general syntactic regularities. Given an annotated training corpus, such methods learn its distributions (the lexical cooccurrences), which requires being given the correct space of events in the mode l tha t is, the grammar-accura te ly enough that they can parse new instances of the same corpus . The success of such models suggests that a statistical model nmst have access to tile appropriate linguistic features to make accurate predictions. We might want to ask the question: what happens if what one wants to do with annotated text is not to annotate more text. but to perform some other task'? Are. the same insights valid, so that annotated text can be used to help in other tasks, for instance generation or translation? Can we use annotated text to investigate properties of language(s) systematically? In other words, can we use annotated text as a repository of information? The answer is a qualified yes. In this paper we look at one type of information that is plentiflflly present in a co rpus syntactic preferences--and we argue that corpora can be very usefifl even for tasks that do not invoh'e parsing directly, but that mak-