An Empirical Evaluation of Models of Text Document Similarity

An Empirical Evaluation of Models of Text Document Similarity Michael D. Lee (michael.lee@adelaide.edu.au) Department of Psychology, University of Adelaide South Australia, 5005, AUSTRALIA Brandon Pincombe (brandon.pincombe@dsto.defence.gov.au) Intelligence Surveillance and Reconnaissance Division, Defence Science and Technology Organisation PO Box 1500, Edinburgh SA 5111 AUSTRALIA Matthew Welsh (matthew.welsh@adelaide.edu.au) Australian School of Petroleum Engineering, University of Adelaide South Australia, 5005,  Modeling the semantic similarity between text docu- ments presents a significant theoretical challenge for cognitive science, with ready-made applications in in- formation handling and decision support systems deal- ing with text. While a number of candidate models exist, they have generally not been assessed in terms of their ability to emulate human judgments of simi- larity. To address this problem, we conducted an ex- periment that collected repeated similarity measures for each pair of documents in a small corpus of short news documents. An analysis of human performance showed inter-rater correlations of about 0.6. We then considered the ability of existing models—using word- based, n-gram and Latent Semantic Analysis (LSA) approaches—to model these human judgments. The best performed LSA model produced correlations of about 0.6, consistent with human performance, while the best performed word-based and n-gram models achieved correlations closer to 0.5. Many of the re- maining models showed almost no correlation with hu- man performance. Based on our results, we provide some discussion of the key strengths and weaknesses of the models we examined.  Modeling the semantic similarity between text docu- ments is an interesting problem for cognitive science, for both theoretical and practical reasons. Theoret- ically, it involves the study of a basic cognitive pro- cess with richly structured natural stimuli. Practically, search engines, text corpus visualizations, and a vari- ety of other applications for filtering, sorting, retriev- ing, and generally handling text rely fundamentally on similarity measures. For this reason, the ability to as- sess semantic similarity in an accurate, automated, and scalable way is a key determinant of the effectiveness of most information handling and decision support soft- ware that deals with text. A variety of different approaches have been devel- oped for modeling text document similarity. These in- clude simple word-based, keyword-based and n-gram measures (e.g., Salton, 1989; Damashek, 1995), and more complicated approaches such as Latent Seman- tic Analysis (LSA: Deerwester et al., 1990; Landauer and Dumais, 1997). While all of these approaches have achieved some level of practical success, they have gen- erally not been assessed in terms of their ability to model human judgments of text document similarity. The most likely reason for this failure is that no suit- able empirical data exist, and considerable effort is in- volved in collecting pairwise ratings of text document similarity for even a moderate number of documents. This paper reports the collection of data that give ten independent ratings of the similarity of every pair of 50 short text documents, and so represents an attempt to establish a ‘psychological ground truth’ for evaluating models. Using the new data, we report a first eval- uation of the ability of word-based, n-gram and LSA approaches to model human judgments.  The text corpus evaluated by human judges contained 50 documents selected from the Australian Broadcast- ing Corporation’s news mail service, which provides text e-mails of headline stories. The documents varied in length from 51 to 126 words, and covered a number of broad topics. A further 314 documents from the same were collected to act as a larger ‘backgrounding’ corpus for LSA. Both document sets were assessed against a stan- dard corpus of five English texts using four models of language. These were the log-normal, generalized in- verse Gauss-Poisson (with γ = −0.5), Yule-Simon and Zipfian models (Baayen, 2001). Both document sets were within the normal range of English text for word frequency spectrum and vocabulary growth and were therefore regarded as representative of normal English texts. Subjects The subjects were 83 University of Adelaide students (29 males and 54 females), with a mean age of 19.7 years. They were each paid with a ten (Australian) dollar gift voucher for every 100 document pair ratings made.