Similarity Between Semantic Spaces

Similarity Between Semantic Spaces Xiangen Hu (xhu@memphis.edu) Department of Psychology; The University of Memphis Memphis, TN 38152 Zhiqiang Cai (zcai@memphis.edu) Department of Psychology; The University of Memphis Memphis, TN 38152 Arthur C. Graesser (agraesser@memphis.edu) Department of Psychology; The University of Memphis Memphis, TN 38152 Matthew Ventura (mventura@memphis.edu) Department of Psychology; The University of Memphis Memphis, TN 38152 Abstract to compute similarities between documents (Hu et al., 2003, 2003; Hu, Cai, Wiemer-Hasting, Graesser, & Mc- Namara, 2005). The issue of evaluating the quality of semantic space is very important, not only at the level of theoretical importance, but also at the level of speci¯c applications. Curren tly, the quality of semantic spaces is usually evaluated by human experts. This is done by compar- ing performances between applications that use a spe- ci¯c semantic space and have human experts perform some benchmark tests. For example, to evaluate an LSA space with a given set of parameters (e.g., num- ber of dimensions), an LSA similarity measure between texts is compared with experts' judgement of the sim- ilarity of those texts (Olde, Graesser, & Tutoring Re- search Group, 2002). There are many possible variables that are involved in creating semantic spaces, which makes it impractical for human experts to evaluate all semantic spaces. In this paper, we present a systematic method to au- tomatically evaluate semantic spaces. This method al- lows us to measure similarity between semantic spaces that are created from di®erent sets of parameters (do- main, corpus size, dimensions, etc.). Furthermore, this metho d can even be used to ¯nd di®erences between se- mantic spaces that are created using entirely di®erent metho ds, such as LSA, HAL, and NLS. One of the challenges in Latent Semantic Analysis (LSA) is deciding which corpus is best for a speci¯c applica- tion.Imp ortant factors of LSA in°uence the generation of high quality LSA space including the size of the cor- pus, the weight (local or global) functions, number of dimensions to keep, etc. These factors are often di±cult to determine and as a result hard to control for. In this paper, we provide a general method to measure simi- larity between semantic spaces. Using this method, one can evaluate semantic spaces (such as LSA spaces) that are generated from di®erent sets of parameters or di®er- ent corpora. The method we have develop ed is generic enough to evaluate di®ering types of semantic spaces. Keywords: Semantic Space, Latent Semantic Analy- sis, Similarity Measures Between Texts Introduction The use of higher dimensional semantic spaces, such as Latent Semantic Analysis (LSA) (Landauer & Littman, 1990; Dumais, 1990; Laham, 1997; Landauer, Laham, Rehder, & Schreiner, 1997; Landauer, Foltz, & La- ham, 1998), Hyperspace Analogue to Language (HAL) (Burgess, Livesay, & Lund, 1996; Burgess & Lund, 1997; Burgess, 1998), Non-Latent Similarity (NLS) algorithm (Cai et al., 2004), is very common in computational lin- guistics. The semantic spaces have been used in applica- tions that involve information retrieval(Dumais, 1990), essay grading, and text comparison (Foltz, Laham, & Landauer, 1999). The scope and depth of the appli- cations are so diverse that di®erent semantic spaces are needed for di®erent purposes (Franceschetti et al., 2001). Furthermore, the process of generating semantic spaces is very complicated (Deerwester et al., 1990). Some of them involve careful selection of corpora (Franceschetti et al., 2001). From all the previous studies and applica- tions of semantic spaces such as LSA, HAL, and NLS, we observed that there are several key parameters (such as dimensions, domain, corpus size, etc.) that need to be set before generating an appropriate space for an ap- plication. There are di®erent metho ds that can be used Observations We ¯rst provide a mathematical model for Semantic Space. This model is simply an abstraction of some commonly used semantic spaces such as LSA , HAL, and NLS. Based on this model, we provide a measure of similarity between semantic spaces. At the end of the paper, we outline pro cedures of how to use the similar- ity measures to evaluate semantic spaces. The metho d presented in this paper is based on the following obser- vations from semantic spaces, such as LSA , HAL, and NLS: 1. Semantics is a property that applies to ¯ve di®er-