An exemplar-based approach to unsupervised parsing

An Exemplar-based Approach to Unsupervised Parsing Simon Dennis (Simon.Dennis@adelaide.edu.au) Department of Psychology; University of Adelaide Adelaide, SA 5005 Australia Abstract issues such as systematicity and constituent for- mation and movement remain unresolved (Hadley, 1994) seriously undermining the empiricist position. In addition, from a practical perspective the in- ability to create syntactic analyses in an unsuper- vised fashion makes the application of natural lan- guage processing systems in new domains tedious. Either one must hand specify appropriate rules or one must create annotated corpora on which to train systems. Both of these tasks are difficult and time consuming. In this paper, we outline attempts to improve an exemplar-based model of unsupervised parsing pro- posed by Dennis (submitted) using span-based nor- malized edit distance (SNED). We start by outlin- ing the exemplar-based approach to parsing. Then we define normalized edit distance and the span- based modification, which is used to align neigh- bours against the target sentence. Then, we dis- cuss how one can calculate the span similarities necessary to apply the method to sentences. Next, we describe a version of Locality Sensitive Hash- ing (Indyk & Motwani, 1998; Gionis et al., 1999) adapted to work with part of speech strings that al- lows the rapid selection of near neighbours. Finally, we present constituent recall and precision data on sentences drawn from the Penn Treebank (Marcus et al., 1993). We present an approach to syntactic processing based on the Syntagmatic Paradignmatic model (Dennis, in press) that assumes that the parse of a sentence can be viewed as a set of alignments with exemplars from memory. Alignment is achieved using a span-based ver- sion of the normalized edit distance measure (Marzal & Vidal, 1993), which is more appropriate for linguistic tasks. Span similarities used in the algorithm are de- rived using a version of the topics model (Griffiths & Steyvers, 2002) in which part-of-speech sequences are generated from their preceeding and postceeding word context. Approximate nearest neighbour exemplars are chosen using Locality Sensitive Hashing (Indyk & Mot- wani, 1998; Gionis, Indyk, & Motwani, 1999). Parses generated by the model are compared against gold stan- dard parses from the Penn Treebank. The method pro- vides state of the art precision and recall on this task and suggests that an unsupervised approach to pars- ing is feasible. Furthermore, the model is more di- rectly comparable to exemplar-based accounts in other areas of cognition such memory and categorization than recursion-based approaches to syntax. Keywords: syntax, parsing, unsupervised, syntagmatic paradigmatic model, edit distance Introduction The nativist/empiricist debate on the origin on lan- guage has been one of the longest and most hotly contested in the history of cognitive science (Pinker, 1994; Elman, 1999). On the one hand, languages are clearly learned at some level with a great many variations that differ in quite subtle ways. Further- more, the difficulty in creating an explanation of how the genes might influence language develop- ment suggests that it is unlikely that our biological endowment has a direct influence (Elman, 1999). However, the fact that humans have a much more complex system of language than other primates, that there are similarities across the world’s lan- guages and that language acquisition takes similar paths in different cultures suggest a strong innate component (Pinker, 1994). One key, if unstated, plank in the nativist case is that to this point no statistical learning procedure capable of capturing the syntax of a complex natu- ral language has been devised (see Dennis, submit- ted; Klein & Manning, 2001). While connectionist models have demonstrated an ability to solve re- stricted problems with toy corpora (Elman, 1991), Exemplar-based Parsing The algorithm that we employ for parsing sentences is a version of the Syntagmatic Paradigmatic model (Dennis, in press, 2004, submitted). The model has been used to account for a number of phenomena including long term grammatical dependencies and systematicity (Dennis, in press), the extraction of statistical lexical information (syntactic, semantic and associative) from corpora (Dennis, 2003a), sen- tence priming (Harrington & Dennis, 2003), verbal categorization and property judgment tasks (Den- nis, in press), serial recall (Dennis, 2003b), and re- lational extraction and inference (Dennis, in press, In this model, sentence parsing involves aligning near neighbour exemplar sentences from memory with the target sentence. For instance, suppose we wish to parse the sentence “His dog was big. ” (see