Part of Speech Tagging Using a Network of Linear Separators

We present an architecture and an on-line learning algorithm and apply it to the problem of part-ofspeech tagging. The architecture presented, SNOW, is a network of linear separators in the feature space, utilizing the Winnow update algorithm. Multiplicative weight-update algorithms such as Winnow have been shown to have exceptionally good behavior when applied to very high dimensional problems, and especially when the target concepts depend on only a small subset of the features in the feature space. In this paper we describe an architecture that utilizes this mistake-driven algorithm for multi-class prediction selecting the part of speech of a word. The experimental analysis presented here provides more evidence to that these algorithms are suitable for natural language problems. The algorithm used is an on-line algorithm: every example is used by the algorithm only once, and is then discarded. This has significance in terms of efficiency, as well as quick adaptation to new contexts. We present an extensive experimental study of our algorithm under various conditions; in particular, it is shown that the algorithm performs comparably to the best known algorithms for POS. 1 I n t r o d u c t i o n Learning problems in the natural language domain often map the text to a space whose dimensions are the measured features of the text, e.g., its words. Two characteristic properties of this domain are that its dimensionality is very high and that both the learned concepts and the instances reside very sparsely in the feature space. In this paper we present a learning algor i thm and an architecture with properties suitable for this domain. The SNOW algorithm presented here builds on recently introduced theories of multiplicative weight-updating learning algorithms for linear functions. Multiplicative weight-updating algorithms such as Winnow (Littlestone, 1988) and Weighted Majority (Littlestone and Warmuth, 1994) have been studied extensively in the COLT literature. Theoretical analysis has shown that they have exceptionally good behavior in the presence of irrelevant attributes, noise, and even a target function changing in t ime (Littlestone, 1988; Littlestone and Warmuth, 1994; Herbster and Warmuth, 1995). Only recently have people started to test these claimed abilities in applications. We address these claims empirically by applying SNOW to one of the fundamental disambiguation problems in natural language: part-of speech tagging. Part of Speech tagging (POS) is the problem of assigning each word in a sentence the part of speech that it assumes in that sentence. The importance of the problem stems from the fact that POS is one of the first stages in the process performed by various natural language related processes such as speech, information extraction and others. The architecture presented here, SNOW, is a Sparse Network Of Linear separators which utilizes the Winnow learning algorithm. A target node in the network corresponds to a candidate in the disambiguation task; all subnetworks learn autonomously from the same data in an online fashion, and at run time, they compete for assigning the correct meaning. A similar architecture which includes an additional layer is described in (Golding and Roth, 1998). The POS problem suggests a special challenge to this approach. First, the problem is a multiclass prediction problem. Second, determining the POS of a word in a sentence may depend on the POS of its neighbors in the sentence, but these are not known with any certainty. In the SNOW architecture, we address these problems by learning at the same time and from the