Using Wordnet to Supplement Corpus Statistics

Data-driven techniques, although commonly used for many natural language processing tasks, require large amounts of data to perform well. Even with significant amounts of data there is always a long tail of infrequent linguistic events, which results in poor statistical estimation. To lessen the effect of these unreliable estimates, we propose augmenting corpus statistics with linguistic knowledge encoded in existing resources. This paper evaluates the usefulness of the information encoded in WordNet for two tasks: improving perplexity of a bigram language model trained on very little data, and finding longer-distance correlations in text. Word similarities derived from WordNet are evaluated by comparing them to association statistics derived from large amounts of data. Although we see the trends we were hoping for, the overall effect is small. We have found that WordNet does not currently have the breadth or quantity of relations necessary to make substantial improvements over purely datadriven approaches for these two tasks. 1 Motivation and Outline Data-driven techniques are commonly used for many natural language processing tasks. However, these techniques require large amounts of data to perform well, and even with significant amounts of data there is always a long tail of infrequent linguistic events. The majority of words, for example, occur only a few times even in a very large corpus. Poor statistical estimation of these rare events will always be a problem when relying on data-driven techniques, especially when only small amounts of data are available. One proposed solution is to augment corpusderived statistics with linguistic knowledge, available in the form of existing lexical and semantic resources. Such resources include lexical databases like WordNet (Fellbaum, 1998), knowledge bases like Cyc (Lenat, 1995), thesauri like Roget’s (Chapman, 1977), and machine readable dictionaries like the Longman Dictionary of Contemporary English (Proctor, 1978). These linguistic resources have been used for many natural language processing tasks, such as resolving syntactic ambiguity (Resnik, 1999), identifying spelling errors (Budanitsky and Hirst, 2001), and disambiguating word senses (Agirre and Rigau, 1996). However, as they are not frequency-based, it is not clear in general how to use them within a statistical framework. In this paper, we consider the use of linguistic knowledge derived from WordNet for combating data sparseness in two language modeling tasks. WordNet is a large, widely-used, general-English semantic network that groups words together into synonym sets, and links these sets with a variety of linguistic and semantic relations. The taxonomic structure of WordNet enables us to automatically derive word similarities, based on variants of a method proposed in (Resnik, 1995). The goal of this paper is to examine the usefulness of these WordNet-derived word similarities for supplementing corpus statistics in the following two tasks. The first task is to improve the perplexity of a bigram language model trained on very little data. If semantically similar words have similar bigram distributions, then for rare words we can use WordNet to find similar, more common proxy words. By combining the bigram data of rare words with their nearest neighbors, we hope to reduce data sparseness and thus better approximate each word’s true bigram distribution. The second task is to find long-distance correlations in text; specifically, we would like to find words that tend to co-occur within a sentence. By identifying these “sticky pairs” we can build language models that better reflect the semantic coherence evident within real sentences. Association statistics collected from data are only reliable, however, for high frequency words. Long-distance associations are generally semantic, and thus WordNet seems an appropriate resource for augmenting the data. In section 2 we describe three measures proposed in the literature for measuring the semantic similarity between concepts in a taxonomy. In addition, we introduce a novel measure of word similarity that takes into account sense frequency. Sections 3 and 4 present a more detailed motivation, methodology, results and error analysis for the bigram and semantic coherence tasks, respectively. Finally, in Section 5 we conclude and discuss future work. 2 Measuring Similarity in a Taxonomy 2.1 Measuring Concept Similarity WordNet is a large, general-English semantic network that represents concepts as groups of synonymous words, called synsets. WordNet is comprised of approximately 110K synsets, which are connected by about 150K edges representing a variety of linguistic and semantic relations. The largest component of WordNet consists of noun synsets connected by hypernym (IS-A) relations, effectively forming a noun taxonomy. The simplest measure of semantic similarity between two concepts in a taxonomy (synsets in WordNet) is the length of the shortest path between them. The shorter the path, the more similar the concepts should be. However, this simple correspondence between path length and similarity is not always valid because edges in a taxonomy often span significantly different semantic distances (Resnik, 1999). For example, in WordNet, eight edges separate rabbit from organism (rabbit IS-A leporid . IS-A mammal . IS-A organism), whereas only one edge separates plankton from organism (plankton IS-A organism). (Resnik, 1995) has proposed an alternative measure of semantic distance in a taxonomy, based on the intuition that the similarity between two concepts is equal to the amount of information they share. He collects counts from a corpus to estimate the probability of each concept in the taxonomy, then uses these probabilities to obtain the information content (negative log likelihood) of a concept. The Resnik similarity between two concepts c1 and c2 is defined as the information content of their least common ancestor (lca): simres(c1, c2) = − log(p(lca(c1, c2)) The Resnik similarity measure has some properties that may not be desirable. For instance, the extent of self-similarity depends on how specific a concept is; two items x and y can be more similar to each other than another item z is to itself. In an attempt to address this and other issues, many other similarity measures have subsequently been proposed. We selected two additional measures suggested in the literature which have shown to be highly correlated with human judgments of similarity. (Jiang and Conrath, 1997) calculate the inverse of semantic similarity – semantic distance. However, since any distance measure can easily be converted into a measure of similarity through simple algebraic manipulation, we still refer to it as a measure of similarity. Although Jiang and Conrath motivate their measure differently, it essentially quantifies the distance between two concepts as the amount of information that is not shared between them. This is just the total amount of information in the two synsets minus their shared information: simjc(c1, c2) = 2 · log(p(lca(c1, c2))) − (log(p(c1)) + log(p(c2))) (Lin, 1998) takes into account both similarities and differences between the two concepts. He normalizes the amount of shared information by the total amount of information, essentially calculating the percentage of information which is shared: simlin(c1, c2) = 2 · log(p(lca(c1, c2))) log(p(c1)) + log(p(c2)) 2.2 Measuring Word Similarity The three similarity measures just described are all designed for determining the similarity between concepts in a taxonomy. However, most words have multiple possible senses corresponding to distinct concepts in the taxonomy. Given a concept similarity measure sim(c1, c2), Resnik defines the similarity between words as the maximum similarity between any of their senses. He searches over all possible combinations of senses and takes the most informative ancestor: wsimmax(w1, w2) = max c1,c2 [sim(c1, c2)] where c1 ranges over the senses of w1 and c2 ranges over the senses of w2. However, as (Resnik, 1999) notes, this method “sometimes produces spuriously high similarity measures for words on the basis of inappropriate word senses. ” In particular, it measures words as highly similar even if it is only their rare senses that are similar in meaning. For instance, wsimmax will find that brazil and pecan are very similar because they are both nuts, which is a rare concept and thus has high information content. However, in broadcast news brazil almost always refers to a country and so on average it is not at all similar to pecan. To address this issue, we devised another word similarity measure, wsimwgt, which is the weighted sum of sim(c1, c2), over all pairs of senses c1 of w1 and c2 of w2, where more frequent senses are weighted more heavily: