Modeling semantic coherence from corpus data: the fact and the frequency of a co-occurrence

One puzzling property of natural language is its so-called infinite productivity, in other words, the ability of a linguistic item to combine with an infinite number of other linguistic items in a meaningful way. From a computational perspective, the task of modeling this phenomenon amounts to estimating the plausibility of the combination of any two or more given linguistic items. Satisfactory solutions to this task are of relevance for various areas in natural language processing: speech recognition, syntactic attachment resolution, word sense disambiguation, information retrieval, etc. Corpus-based models are particularly capable of accomplishing this task: using no other data than corpora they can automatically form a representation of a given item, generalize it to a linguistic (morphological, syntactic, semantic) category and later handle the item using available knowledge about the category. The present paper addresses problems of estimating semantic coherence between words in a phrase. Out of the corpus-based models, the so-called similarity-based models display a better ability to account for finer semantic distinctions, rather than thesaurus-based models. The latter makes use of predefined semantic classes of words, such as synonym groups in WordNet (e.g. Li and Abe 1998). This approach assumes that all words within a class have similar co-occurrence features and that the classes have clear-cut boundaries. This assumption is reasonable when only crude semantic distinctions are sufficient, for example, in the task of syntactic attachment resolution. However, if one is concerned with finer semantic distinctions, this approach is not adequate, as a word, on the one hand, has semantic dissimilarities with other words within its class and, on the other hand, can be semantically close to words of other classes (Schutze 1997). The similarity-based approach does not use any precompiled semantic classes; instead, they are computed from corpus data and therefore they display a gradient character1. The gradient character of the thus derived category is cognitively plausible: it reflects the fact that people display different degrees of certainty about the appropriateness of a combination of a word with different members of the same category. The general design of a similarity-based model of a word-co-occurrence can be described as follows. A word w is represented in terms of words wi1.n with which w j co-occurs. For deriving a representation of w, the application searches through the corpus for its occurrences and counts words occurring in the context of w. To characterize the context of w, two strategies have been tried: the window-based and the syntactic. In the first case the context is marked out by imposing a window of a certain size around w (e.g., Gale, Church, and Yarowsky (1992) used a thousand-word window) The other strategy is to limit context to words appearing in a certain syntactic relation to w, such as direct objects of a verb (Grefenstette 1996; Pereira, Tishby, and Lee 1993). After the data are gathered, the semantics of w is represented in terms of a vector in an n-dimensional