Kernels Incorporating Word Positional Information in Natural Language Disambiguation Tasks

In this paper, we introduce a new kernel function designed for the problem of word sense disambiguation. The presented kernel function employs two different types of positional information related to words present in the contexts of the words to be disambiguated. For each pair of words in two contexts, the proposed kernel takes into account both their distances from the ambiguous words and also the difference of their mutual positions. We apply the kernel to contextsensitive spelling correction with SVMs and show that it significantly outperforms other considered kernels. Introduction Many natural language processing applications require accurate resolution of the various kinds of ambiguity present in natural language, giving rise to a class of disambiguation problems. In this paper, we focus on lexical disambiguation problems, where disambiguation is done at the level of words. A common example of such a problem is word sense disambiguation (WSD), where the task is to resolve the correct sense for an instance of a polysemous word, for example, the word bank where the ambiguity is between the senses “river bank” and “financial institution”. A lexical disambiguation problem closely related to WSD is context-sensitive spelling error correction, where the misspelling of the original word belongs to the language, such as, for example, desert misspelled as dessert. This mistake cannot be detected by standard lexicon-based checkers, since dessert belongs to the English lexicon. A set of similar words that belong to the lexicon and that are often confused with the other words in the set is called a confusion set. For example, {piece, peace} can be considered as a binary confusion set. Lexical disambiguation problems other than WSD can also be used as alternatives for the purpose of evaluating WSD systems. For example, (Yarowsky 1994) studies the problem of restoring accents in Spanish and French texts as a substitute for the WSD problem. It is easy to cast context-sensitive spelling error correction as a WSD problem such that each word of the confusion set is considered as a “sense”. A common motivation to use a substitute problem for the WSD is that data necessary for evaluating the Copyright c © 2005, American Association for Artificial Intelligence (www.aaai.org). All rights reserved. methods can be obtained automatically for these problems, whereas data for the WSD problem have to be annotated by hand. In order to disambiguate the sense of an ambiguous word, any WSD method has to incorporate the information about its context, that is, the words surrounding it in the text. A common way of representing the context is a bag-of-words (BoW), where no information of the positions of the words in the context is preserved. Positional information is directly incorporated by the ordered BoW model (Audibert 2004), where each word of the context is represented together with its position and thus two occurrences of one word at different positions in the context are considered different features. Previously, we introduced a weighted BoW approach, where the context words are weighted in such a way that the words closer to the ambiguous word receive higher values, motivated by the assumption that closer words are more relevant for disambiguation (Ginter et al. 2004). We present here a new kernel function that generalizes the aforementioned BoW approaches. The proposed WSD kernel further incorporates a new type of positional information not present in the previously introduced BoW approaches. Apart from the weight associated with a context word’s absolute distance from the ambiguous word itself, the proposed kernel also takes into account for two words in the compared contexts the mutual difference in the words’ positions relative to the ambiguous word. We evaluate the method using Support Vector Machines (SVMs), because they have been repeatedly shown to provide state-of-the-art performance in natural language disambiguation tasks. We use context-sensitive spelling error correction as the model problem and show a significant gain in performance compared to the weighted BoW and to the standard BoW approach. Binary Classification with Support Vector Machines We begin with a short description of SVMs. A more comprehensive introduction can be found, for example, in (Rifkin 2002; Vapnik 1998). In a binary classification task, the training data is comprised ofm labeled examples (x1, y1), . , (xm, ym), where xi ∈ X are training data points and yi ∈ {−1,+1} are the corresponding class labels. SVMs can be considered as a special case of the following regularization problem known as Tikhonov regularization: