From Social Networks To Distributional Properties: A Comparative Study On Computing Semantic Relatedness

From Social Networks To Distributional Properties: A Comparative Study On Computing Semantic Relatedness Ulli Waltinger (ulli marc.waltinger@uni-bielefeld.de) Text Technology, Bielefeld University Irene Cramer (irene.cramer@udo.edu) Faculty of Cultural Studies, TU Dortmund University Tonio Wandmacher (tonio.wandmacher@uni-osnabrueck.de) Institute of Cognitive Science, University of Osnabr¨uck Abstract Introduction behaving similarly. For example they can show a strong asso- ciative relationship (e.g. ball - goal), and they can be related across different linguistic categories (e.g. milk - white, dog - bark). With respect to the automatic computation of SR, how- ever, many research questions remain unanswered. As stated above, many algorithms were presented in the past decade, but thorough evaluations and comparisons of their ability to capture SR in a human-like manner are still rare. In this work we therefore present a study comparing various semantic re- latedness measures. We evaluate sixteen different algorithms involving four different resources based on a human judge- ment experiment, and we analyze the algorithms from a the- oretical and practical point of view. The paper is organized as follows: the subsequent section describes two works rep- resenting the methodological basis for our study. The various semantic relatedness measures employed in our experiment are described in Section Semantic Relatedness Measures. The experimental setup as well as the results obtained are pre- sented in Section Evaluation. The computation of semantic relatedness (SR) has become an important task in many NLP applications such as spelling error detection, automatic summarization, word sense dis- ambiguation, and information extraction. In recent years a large variety of approaches in computing SR has been pro- posed. However, algorithms and results differ depending on resources and experimental setup. It is obvious that SR plays a crucial role in the lexical retrieval of humans. In various priming experiments it could be shown that semantically re- lated terms influence the semantic processing of one another. For example, if ”bread” is primed by ”butter” it is recog- nized more quickly. Moreover, many theories of memory are based on the notion of SR. The spreading activation theory of (Collins & Loftus, 1975) for example groups lexical items according to their SR in a conceptual graph. Similar ideas can be found in the ACT theory of Anderson (Anderson, 1983). The question that arises for us is, how this kind of relatedness can be determined by automatic means. In the literature the notion of SR is often confounded with semantic similarity; there is however a clear distinction between these notions. Two terms are semantically similar if they behave similarly in a given context and if they share some aspects of meaning (e.g. in the case of synonyms or hypernyms). On the other hand two terms can be semantically strongly related without The task of estimating SR between two given lexical items can be performed by humans in an effortless and intuitive manner. However, this notion is very difficult to formalize from a psycholinguistic or computational point of view. In terms of an evaluation of SR algorithms, most commonly human judgement experiments are conducted. The perfor- mance of an SR measure is determined by directly compar- ing the automatic computed results with those gained from the human judgements via correlation. As a most prominent example Budanitsky and Hirst (Budanitsky & Hirst, 2006) presented a comparison of five semantic relatedness mea- sures for the English language. They recommended a three- level evaluation including theoretical examination, compari- son with human judgements and evaluation with respect to a given NLP-application. The measures were evaluated on two different data sets: The first data set was compiled by Ruben- stein and Goodenough (Rubenstein & Goodenough, 1965); it contained 65 word-pairs. The second set, containing 30 word pairs, was compiled by Miller and Charles (Miller & Charles, 1991). For each of the five measures, Budanitsky and Hirst reported correlation coefficients between 0.78 and 0.83. Boyd-Graber et al. (Boyd-Graber, Fellbaum, Osher- In recent years a variety of approaches in computing seman- tic relatedness have been proposed. However, the algorithms and resources employed differ strongly, as well as the results obtained under different experimental conditions. This article investigates the quality of various semantic relatedness mea- sures in a comparative study. We conducted an extensive ex- periment using a broad variety of measures operating on so- cial networks, lexical-semantic nets and co-occurrence in text corpora. For two sample data sets we obtained human relat- edness judgements which were compared to the estimates of the automated measures. We also analyzed the algorithms im- plemented and resources employed from a theoretical point of view, and we examined several practical issues, such as run time and coverage. While the performance of all measures is still mediocre, we could observe that in terms of of coverage and correlation distributional measures operating on controlled corpora perform best. Keywords: Semantic Relatedness; Semantic Similarity; Hu- man Judgement; Social Networks; WordNet; LSA; Related Work