Adding syntactic information to LSA

Adding syntactic information to LSA Peter Wiemer-Hastings Peter.Wiemer-Hastings@ed.ac. uk School of Cognitive Science / ICCS, Division of Informatics University of Edinburgh, Edinburgh EH8 9LW Scotland Abstract Much eﬀort has been expended in the ﬁeld of Natural Language Understanding in developing methods for de- riving the syntactic structure of a text. It is still unclear, however, to what extent syntactic information actually matters for the representation of meaning. LSA (La- tent Semantic Analysis) allows you to derive informa- tion about the meaning without paying attention even to the order of words within a sentence. This is consis- tent with the view that syntax plays a subordinate role for semantic processing of text. But LSA does not per- form as well as humans do in discriminating meanings. Can syntax be the missing link that will help LSA? This paper seeks to address that question. Introduction In the beginning, there was syntax. And it was good. But it did not give us what we really want to know about a text — what it means. Then there was latent seman- tic analysis (Deerwester, Dumais, Furnas, Landauer, & Harshman, 1990, LSA), which provided a means of com- paring the “semantic” similarity between a source and target text, and thereby giving some idea of meaning of the source. That was good too, almost as good as hu- mans in a simple task, but not quite. Because LSA pays no attention to syntax at all — not even word order — one promising approach to improving LSA is by giving it some of the information that is provided by syntax. Knowledge about the syntactic structure of a sentence provides information about the relationships between the words: which words modify which other words, and the relationships between verbs and their arguments or the- matic roles. The research presented here is an attempt to evaluate the beneﬁts of providing LSA with thematic role information which comes from syntactic knowledge. Previous work The primary goal of the AutoTutor project (Graesser, Franklin, Wiemer-Hastings, & the Tutoring Research Group, 1998; Wiemer-Hastings, Wiemer-Hastings, & Graesser, 1999) is to model human tutorial dialogue. It is based on studies of the discourse patterns of hu- man tutors during tutoring sessions (Person, Graesser, Magliano, & Kreuz, 1994). These analyses have shown that human tutors do not have complete understanding of their students’ answers to questions, but the do get an approximation. For AutoTutor, LSA provides such approximate understanding of student inputs by com- paring them to expected answers, and using the LSA cosines as a metric of the extent to which the student entered what was expected. We evaluated this approach by randomly selecting a set of 8 student answers to each of 24 questions in our do- main of computer literacy (Wiemer-Hastings, Graesser, Harter, & the Tutoring Research Group, 1998). We asked human raters to evaluate these answers by pro- viding an aggregate measure of the percentage of stu- dent answer propositions that “match” some expected answer proposition. Proposition was deﬁned loosely as an atomic sentence. Match was left to the human raters to deﬁne. Then we performed the same analysis with LSA, modeling the match function by adjusting the co- sine threshold. The best performance was realized with a 200-dimensional space with a cosine threshold of 0.5. This provided a correlation of r = 0.49 with the average rating of the human judges. Because the distribution of ratings was skewed, we also calculated Cronbach’s al- pha. The average alpha score between human raters was α = 0.76. The alpha score between LSA and the aver- age human rating was α = 0.60. These results were very encouraging. LSA provided much of the discrimination shown by human raters, enough to use in the AutoTutor system. It could however, be improved. The obvious information source that LSA ignores is syntax. It is a “bag-of-words” approach, simply adding together term vectors to make a vector for a text. This paper is an attempt to identify whether the addition of syntactic knowledge can strengthen LSA judgments. Related work Partially as a result of the Behaviorist movement in psy- chology, linguistics and natural language processing fo- cused for a long time primarily on the syntactic structure of sentences (Chomsky, 1981, for example). In the 70’s and 80’s, Schank sought to change this by claiming that semantics alone was suﬃcient (Schank & Riesbeck, 1981, for example). More recently, researchers from psychol- ogy have championed LSA as both a technique for de- termining the meaning of texts and as a model of human language. Much of the recent interest (and controversy) regard- ing LSA can be traced to Landauer, Kintsch, and col- leagues. They imported LSA from the realm of infor- mation retrieval and hailed it as part or parcel of a psy-