Using Latent Semantic Analysis to Estimate Similarity

Using Latent Semantic Analysis to Estimate Similarity Sabrina Simmons (s.g.simmons@warwick.ac.uk) Department of Psychology, University of Warwick Coventry CV4 7AL, UK Zachary Estes (z.estes@warwick.ac.uk) Department of Psychology, University of Warwick Coventry CV4 7AL, UK Abstract In three studies we investigated whether LSA cosine values estimate human similarity ratings of word pairs. In study 1 we found that LSA can distinguish between highly similar and dissimilar matches to a target word, but that it does not reliably distinguish between highly similar and less similar matches. In study 2 we showed that, using an expanded item set, the correlation between LSA ratings and human similarity ratings is both quite low and inconsistent. Study 3 demonstrates that, while people distinguish between taxonomic / thematic word pairs, LSA cosines do not. Although people rate taxonomically related items to be more similar than thematically related items, LSA cosine values are equivalent across stimuli types. Our results indicate that LSA cosines provide inadequate estimates of similarity ratings. Latent Semantic Analysis (LSA) is a statistical model of language learning and representation that uses vector averages in semantic space to assess the similarity between words or texts (Landauer & Dumais, 1997). LSA has been used to model a number of cognitive phenomena and correlates well with many human behaviors relating to language use. Owing to the model’s success, LSA ratings are now being used in place of human ratings as measures of semantic similarity. The purpose of this paper is to investigate whether LSA cosine values are adequate substitutes for human semantic similarity ratings. recording the frequency that a given word occurs in particular texts, the model weights entries to reflect the diagnosticity of a word for a given context. For example, a word that appears in a large number of very different contexts is not as diagnostic as a word that occurs less often and only in a small set of similar contexts. The next steps in the process are essential to LSA’s ability to uncover higher order associations between words. Through a combination of singular value decomposition (SVD) and dimension reduction, the representations of words that occur in the same or similar contexts become themselves, more similar (Kwantes, 2005). In this case, a word’s representation is a vector in semantic space that summarizes information about the contexts in which that word is found. In this way the similarity between two words can be determined by the cosine between their vectors (although other methods are sometimes used, see Rehder, Schreiner, Wolfe, Laham, Landauer, & Kintsch, 1998). Thus, through a process that contains no information about semantic features, the definition of words, word order, or parts of speech, LSA is able to capture subtle relationships between words that might never have occurred together (Landauer & Dumais, 1997). LSA in Application Overview of LSA LSA is able to model several human cognitive abilities and has a number of potential practical applications. To give a few examples, LSA can imitate the vocabulary growth rate of a school age child (Landauer & Dumais, 1997), is able to recognize synonyms about as accurately as prospective college students who speak English as a second language (Landauer & Dumais 1994), can pass a college level multiple choice test in psychology (Landauer, Foltz, & Laham, 1998), can successfully simulate semantic priming (Landauer & Dumais, 1997), assesses essay quality in a manner consistent with human graders (Landauer, et. al. 1998), and is able to determine what text a student should use in order to optimize learning (Wolfe, Schreiner, Rehder, Laham, Foltz, Kintsch & Landauer, 1998). Because of its success at modeling human performance on such a range of semantic tasks, LSA is sometimes used as a tool for stimuli norming and construction. For example, The basic theory behind LSA is that the “psychological similarity between any two words is reflected in the way they co-occur in small sub-samples of language” (Landauer & Dumais, 1997 p. 215). The model begins with a matrix taking words as rows and contexts as columns, although, theoretically, many other types of information could also be used. The contexts may be anything, for example, newspaper articles, textbooks, or student essays and the words are simply those that appear in the training set. Importantly, the contexts with which the model is provided will determine what types of words it has experience with, so the training set should be relevant to the task the model is to perform. The first step is to associate each word with the contexts in which it is likely to appear. In addition to