Rules for Syntax, Vectors for Semantics

Rules for Syntax, Vectors for Semantics Peter Wiemer-Hastings (Peter.Wiemer-Hastings@ed.ac.uk) Iraide Zipitria (iraidez@cogsci.ed.ac.uk) University of Edinburgh Division of Informatics 2 Buccleuch Place Edinburgh EH8 9LW Scotland Abstract Latent Semantic Analysis (LSA) has been shown to perform many linguistic tasks as well as humans do, and has been put forward as a model of human linguistic competence. But LSA pays no attention to word order, much less sentence structure. Researchers in Natural Language Processing have made significant progress in quickly and accurately deriv- ing the syntactic structure of texts. But there is little agree- ment on how best to represent meaning, and the representa- tions are brittle and difficult to build. This paper evaluates a model of language understanding that combines information from rule-based syntactic processing with a vector-based se- mantic representation which is learned from a corpus. The model is evaluated as a cognitive model, and as a potential technique for natural language understanding. Motivations Latent Semantic Analysis (LSA) was originally developed for the task of information retrieval, selecting a text which matches a query from a large database (Deerwester, Du- mais, Furnas, Landauer, & Harshman, 1990) 1 . More re- cently, LSA has been evaluated by psychologists as a model for human lexical acquisition (Landauer & Dumais, 1997). It has been applied to other textual tasks and found to gen- erally perform at levels matching human performance. All this despite the fact that LSA pays no attention to word or- der, let alone syntax. This led Landauer to claim that syntax apparently has no contribution to the meaning of a sentence, and may only serve as a working memory crutch for sen- tence processing, or in a stylistic role (Landauer, Laham, Rehder, & Schreiner, 1997). The tasks that LSA has been shown to perform well on can be separated into two groups: those that deal with sin- gle words and those that deal with longer texts. For exam- ple, on the synonym selection part of the TOEFL (Test of English as a Foreign Language), LSA was as accurate at choosing the correct synonym (out of 4 choices) as were successful foreign applicants to US universities (Landauer et al., 1997). For longer texts, Rehder et al (1998) showed that for evaluating author knowledge, LSA does steadily worse for texts shorter than 200 words. More specifically, 1 We do not describe the functioning of the LSA mechanism here. For a complete description, see (Deerwester et al., 1990; Landauer & Dumais, 1997) for 200-word essay segments, LSA accounted for 60% of the variance in human scores. For 60-word essay segments, LSA scores accounted for only 10% of the variance. In work on judging the quality of single-sentence student answers in an intelligent tutoring context, we have shown in previous work that although LSA nears the performance of intermediate-knowledge human raters, it lags far behind expert performance (Wiemer-Hastings, Wiemer-Hastings, & Graesser, 1999b). Furthermore, when we compared LSA to a keyword-based approach, LSA performed only marginally better (Wiemer-Hastings, Wiemer-Hastings, & Graesser, 1999a). This accords with unpublished results on short answer sentences from Walter Kintsch, personal com- munication, January 1999. In the field of Natural Language Processing, the eras of excessive optimism and ensuing disappointment have been followed by study increases in the systems’ ability to pro- cess the syntactic structure of texts with rule-based mecha- nisms. The biggest recent developments have been due to the augmentation of the rules with corpus-derived proba- bilities for when they should be applied (Charniak, 1997; Collins, 1996, 1998, for example). Unfortunately, progress in the area of computing the se- mantic content of texts has not been so successful. Two ba- sic variants of semantic theories have been developed. One is based on some form of logic. The other is represented by connections within semantic networks. In fact, the latter can be simply converted into a logic-based representation. Such theories are brittle in two ways. First, they require every concept and every connection between concepts to be defined by a human knowledge engineer. Multi-purpose representations are not feasible because of the many techni- cal senses of words in every different domain. Second, such representations can not naturally make the graded judge- ments that humans do. Humans can compare any two things (even apples and oranges! ), but aside from count- ing feature overlap, logic-based representations have diffi- culty with relationships other than subsumption and “has- as-part”. Due to these various motivations, we are pursuing a two- pronged research project. First, we want to evaluate the combination of a syntactic processing mechanism with an LSA-based semantic representation as a cognitive model of human sentence similarity judgements. Second, we are